New Page 1

Desired Features

The aim of the new world order build system is to separate the functionality of the build system frame work and the component related build scripts. This is achieved by having a clear semantics on how the component related scripts will be invoked.

It should have following features –

Easy of use.
Easy of adding new components without modifying the build system code.
Easy of adding new platforms preferable without modifying the build system code.
Building multiple components at the same time if they are independent and follow the dependencies when required.
Log reporting in xml format.
Log the build related information in the database.
Should transfer the required externals for a particular component to the remote build machine. This can even include externals like Java. Flexibility on this feature is debatable.
Jobs that failed and go to idle should never be re-matched to the same host.
Build system should be smart enough to handle the operating system signals and should report failure if any of the scripts is killed.
Web interface for component builders to build over the web.

Semantics

The build system functionality is divided into two

Tasks done at the submit machine
Task done at the remote machine

These semantics are shown below. Lines written in "blue" are the functionalities provided by the build system. Lines written in "black" are the "Glue-Scripts" provided by the component builder (user). Glue-Scripts bind the two entities (build system scripts and component related scripts) together. Build system framework expects specific code of conduct from the Glue-Scripts but is not concerned with how they achieve their tasks. Lines in "red" are remarks or more information available to the user.

Submit-side Semantics

Submit side of the build system is mainly concerned with doing the preliminary tasks done before the component build is actually dispatched to remote machines and doing the post build tasks after the component build results arrive back from the remote machine.

User initiates the build by passing the component description file to the build system. This component description file has all the information that the build system requires to successfully build the component. A sample component description file is shown below. It contains the information about the component and where the glue scripts are and what glue script to invoke at a given time. Once this information is parsed by the build system it creates an initial database record for this run and gets a runid. It also creates a buildspace/runspace where the build related information is stored. It creates the required sub-directories based on the platform list. Once the stage is set build system fetches component sources as instructed in the component description file and places them in a known location. It then invokes the pre_all script for the component if declared. Once the general tasks are done build system creates condor submit file for the requested platforms and submits a condor job that runs on the remote machine. Build system also invokes the platform_pre and platform_post scripts if declared at appropriate time to complete tasks related to the component build.

Once the condor job completes (successfully or unsuccessfully) build system invokes the post_all script. It then updates the database with the result status of each task.

Read the component description file
Create an initial database record and get a BuildID
Fetch sources                            ..............
pre_all                                  ..............
for each platform {
    cp common/* <arch>/
    cd <arch>
    platform_pre                         ..............
    submit vanilla job
    update database
    platform_post                        ..............
    update_db
}
post_all                                 ..............
update database
display html

Fetches sources in $Workspace/$BuildID/SOURCES
Wakes up in $Workspace/$BuildID/

Wakes up in $Workspace/$BuildID/$platform

Wakes up in $Workspace/$BuildID/$platform

Wakes up in $Workspace/$BuildID/

Execute-side Semantics (per platform)

Once the condor job (job to build the component on a targeted platform) comes to the execute machine, the very first script to be invoked is the build system wrapper script. After this the user is given a chance to split the build into several tasks of his/her choice in the remote_declare. If the user needs to take any action before declaring the remote tasklist he/she can do it in remote_pre_declare. Similarly if there are any tasks to be executed before the first build task but after the remote_declare, this can be done in remote_pre.

Then for every task declared by the user the build system runs them saves the statistics and results for the task and updates the database. When the component is built and all the tasks are completed user can declare a remote_post script. This essentially serves for the tasks like packaging the component in a particular way which is not a part of the build but is required for component distribution. When everything is done the results of all the tasks are sent back to the submit side.

Component build wrapper
remote_pre_declare                       ..............
remote_declare                           ..............
remote_pre                               ..............
if declare_list is empty {
    insert a special noop task in the tasklist
}
for each task in declare_list {
    remote_task                          ..............
    record task runtime
    save error and output
    save return status
    update_db
}
remote_post                              ..............
send back all results

Component Description File

[parag@vger build]$ cat condor.desc
description = nightly condor 6.6 test run

project = condor
project_release = 6, 6, 7

component = condor
component_version = 6, 6, 7

sources = condor_V6_6-branch.cvs, condor-builder-R5.cvs

pre_all = pre_all.sh
pre_all_args = foo bar baz
platform_pre = platform_pre.sh
remote_pre_declare = pre_declare.sh
remote_declare = declare.sh
remote_declare_args = quick
remote_pre = pre.sh
remote_task = task.sh
remote_post = post.sh
platform_post = platform_post.sh
post_all = post_all.sh

platforms = i386_rh9, sun4x_58

notify = condor-staff@cs.wisc.edu, pfc@cs.wisc.edu

priority = 1

[parag@vger build]$ cat condor_V6_6-branch.cvs
#this is a comment
method=cvs
cvs_root = :ext:parag@chopin.cs.wisc.edu:/p/condor/repository/CONDOR_SRC
cvs_tag = BUILD-V6_6-branch-2004-6-24
cvs_module = V6_6_BUILD
#cvs_source_path = nmi_build/std_contrib

[parag@vger build]$ cat condor-builder-R5.cvs
#this is a comment
method=cvs
cvs_root=:ext:parag@chopin.cs.wisc.edu:/p/condor/repository/nmi
cvs_tag = nmi_r5_branch
cvs_module=R5_COMMON
#cvs_source_path = nmi_build/std_contrib
[parag@vger build]$

Description of parameters used in the component description file

Parameters	Description
description	Component description string
project	Name of the project
project_release	Version of the Project
component	Name of the component
component_version	Version of the Component
sources	List of sources to be fetched
platforms	List of platforms to build on
notify	Email notification list
priority	???
...	...

Sample Glue scripts

pre_all

[firebird.cs.wisc.edu:build] cat pre_all
#!/usr/bin/env perl

######################################################################
# $Id: pre_all,v 1.1.2.4 2004/06/26 03:34:53 wright Exp $
# Submit-side NMI build system infrastructure to prepare sources for
# the Condor build
#
# We assume the 'common' subdir contains everything we checked out of
# CVS. We need to go into that directory, and do a few things:
# 1) Figure out what version of Condor we're building
# 2) run autoconf so we have a working configure script
# 3) build the nroff source for the man pages so that the man-current
# external package will work
# 4) Setup the sources as a release-able source tarball
######################################################################

######################################################################
# NOTE: This script depends on the following bootstrap programs:
# - GNU tar (so long as it supports --exclude)
# - GNU autoconf (version 2.57 or greater)
# - everything require for "make nroff"
# - latex
# - latex2html
# ... ;) (could fill this in later)
######################################################################

use Cwd;

# autoflush our STDOUT
$| = 1;

my $CommonDir = getcwd();
my $SrcDir = "$CommonDir/src";
my $DocDir = "$CommonDir/doc";
my $ExtDir = "$CommonDir/externals";

-d $CommonDir || die "$CommonDir does not exist!\n";
-d $SrcDir || die "$SrcDir does not exist!\n";
-d $DocDir || die "$DocDir does not exist!\n";
-d $ExtDir || die "$ExtDir does not exist!\n";

######################################################################
# 1) Figure out what version of Condor we're building
######################################################################

print "Figuring out Condor version\n";
chdir( "$CommonDir" ) || die "Can't chdir($CommonDir): $!\n";
my $vers_src="$SrcDir/condor_c++_util/condor_version.C";
my $vers_file="CONDOR-VERSION";
my $majorv;
my $minorv;
my $releasev;
my $version;

open( VERS_SRC, "<$vers_src" ) || die "Can't open $vers_src: $!\n";
while( <VERS_SRC> ) {
if( /^static char\* CondorVersionString = \"\$CondorVersion: (\d+)\.(\d+)\.(\d+).*/ ) {
$majorv = $1;
$minorv = $2;
$releasev = $3;
$version = "$majorv.$minorv.$releasev";
last;
}
}
close( VERS_SRC );
if( $version =~ /(\d+)\.(\d+)\.(\d+)/ ) {
print "Condor version is: $version\n";
} else {
die "ERROR: Can't parse the version string!";
}
open( VERS_FILE, ">$vers_file") || die "Can't open $vers_file: $!\n";
print VERS_FILE "$version";
close( VERS_FILE );

######################################################################
# Step #2: run autoconf
######################################################################

print "Running autoconf to generate configure script\n";
chdir( "$SrcDir" ) || die "Can't chdir($SrcDir): $!\n";
system( "autoconf" );
$ac_status = $?;
if( $ac_status ) {
die "Autoconf failed with status $ac_status\n";
}
system( "rm -rf autom4te.cache" );
$status = $?;
if( $status ) {
warn "Can't remove autom4te.cache directory!\n";
}
print "Finished running autoconf\n";

######################################################################
# Step #3: build nroff sources
######################################################################

print "Building man page nroff source\n";
chdir( "$DocDir" ) || die "Can't chdir($DocDir): $!\n";
my $doc_log = "$CommonDir/nroff.log";
open( DOC, "make nroff 2>&1|" ) || die "Can't open 'make nroff': $!\n";
open( DOC_LOG, ">$doc_log" ) || die "Can't open $doc_log: $!\n";
$oldfh = select(DOC_LOG); $| = 1; select($oldfh);
while( <DOC> ) {
print DOC_LOG "$_";
}
close( DOC );
close( DOC_LOG );
$nroff_status = $?;
if( $nroff_status ) {
die "Building nroff man pages failed with status $nroff_status!\n";
} else {
unlink( $doc_log ) || warn "Can't unlink($doc_log): $!\n";
}
system("tar zcf $ExtDir/bundles/man/current/man-current.tar.gz man");
print "Finished building man page nroff source\n";

chdir( "$CommonDir" ) || die "Can't chdir($CommonDir): $!\n";
print "Removing doc directory\n";
system( "rm -rf $DocDir" );
print "Done removing doc directory\n";

######################################################################
# 4) Setup the sources as a release-able source tarball
######################################################################

print "Creating source tarball\n";
chdir( "$CommonDir" ) || die "Can't chdir($CommonDir): $!\n";
my $src_tar_dir = "condor-$version";
my $src_tarball = "$src_tar_dir.tar.gz";
mkdir( "$src_tar_dir" ) || die "Can't mkdir($src_tar_dir): $!\n";

safe_sys( "cp $SrcDir/README.building $src_tar_dir" );
safe_sys( "cp src/condor_release/LICENSE.TXT $src_tar_dir" );
safe_sys( "mv NTconfig config imake src externals $src_tar_dir" );
safe_sys( "tar --exclude=CVS -zcf $src_tarball $src_tar_dir" );
print "Done creating source tarball\n";

print "Removing all source directories\n";
safe_sys( "rm -rf $src_tar_dir" );
print "Done removing all source directories\n";

print "All steps completed successfully!\n";
exit 0;

######################################################################
# subroutines
######################################################################

sub safe_sys() {
my ( $arg ) = @_;
system $arg;
my $status = $?;
if( $status ) {
die "$arg failed with status $status\n";
}
}
[firebird.cs.wisc.edu:build]

remote_declare

[firebird.cs.wisc.edu:build] cat remote_declare
#!/usr/bin/env perl

# $Id: remote_declare,v 1.1.2.1 2004/06/25 04:05:01 wright Exp $

# Condor implementation of NMI build infrastructure interface.
# This is just a trivial hard-coded declare list, which is the
# same on all platforms, all versions, etc.

open( LIST, ">tasklist.nmi" ) || die "Can't open tasklist.nmi: $!\n";
print LIST "externals\n";
print LIST "release\n";
print LIST "static\n";
print LIST "public\n";
close( LIST );
[firebird.cs.wisc.edu:build]

remote_declare

remote_pre

[firebird.cs.wisc.edu:build] cat remote_pre
#!/usr/bin/env perl

######################################################################
# $Id: remote_pre,v 1.1.2.2 2004/06/25 05:15:51 wright Exp $
# script to setup the Condor build
######################################################################

use Cwd;
my $BaseDir = getcwd();
my $SrcDir = "$BaseDir/src";

# autoflush our STDOUT
$| = 1;

######################################################################
# run configure on the source tree
######################################################################

chdir( $SrcDir ) || die "Can't chdir($SrcDir): $!\n";
print "running CONFIGURE ...\n";
open( CONFIG, "./configure 2>&1 |") ||
die "Can't open configure as a pipe: $!\n";
while ( <CONFIG> ) {
print $_;
}
close( CONFIG );
$configstat = $?;
print "CONFIGURE returned a code of $configstat\n";
($configstat == 0) || die "CONFIGURE failed, aborting build\n";

[firebird.cs.wisc.edu:build]

remote_task

[firebird.cs.wisc.edu:build] cat remote_task
#!/usr/bin/env perl

######################################################################
# $Id: remote_task,v 1.1.2.2 2004/06/25 05:15:51 wright Exp $
# Perform a given build task, and return the status of that task
# 0 = success
# 1 = build failed
# 3 = internal fatal error (a.k.a. die)
######################################################################

use Cwd;
my $BaseDir = getcwd();
my $SrcDir = "$BaseDir/src";

# autoflush our STDOUT
$| = 1;

######################################################################
###### WARNING!!! The return value of this script has special ######
###### meaning, so you can NOT just call die(). you MUST ######
###### use the special c_die() method so we return 3!!!! ######
######################################################################

if( ! defined $ENV{_NMI_TASKNAME} ) {
c_die("_NMI_TASKNAME not in environment, can't build anything!\n");
}
my $taskname = $ENV{_NMI_TASKNAME};
if( ! $taskname ) {
# if we have no task, just return success immediately
print "No tasks specified, returning SUCCESS\n";
exit 0;
}

######################################################################
# build the task
######################################################################

chdir("$SrcDir") || c_die("Can't chdir($SrcDir): $!\n");
open( BUILD, "make $taskname 2>&1 |" ) ||
c_die("Can't run make $taskname\n");
while( <BUILD> ) {
print $_;
}
close( BUILD );
$buildstatus = $?;
if( $buildstatus == 0 ) {
print "Building $taskname: SUCCESS\n";
exit 0;
}

print "Building $taskname: FAILURE ($buildstatus)\n";
if( $taskname eq "externals" && -d "$BaseDir/externals/build" ) {
# if the externals failed to build, we want to grab a copy of all
# the build logs from the externals tree into our public output
# directory. however, we only care if we got far enough into the
# externals build process to have any build logs
if( ! -d "$BaseDir/public" ) {
mkdir( "$BaseDir/public" ) ||
c_die("Can't mkdir($BaseDir/public): $!\n");
}
$pub_ext_log = "$BaseDir/public/external-logs";
if( ! -d "$pub_ext_log" ) {
mkdir( "$pub_ext_log" ) ||
c_die("Can't mkdir($pub_ext_log): $!\n");
}
$ext_build = "$BaseDir/externals/build";
opendir( DIR, "$ext_build" ) ||
c_die("Failed to opendir($ext_build): $!\n");
@files = readdir(DIR);
closedir DIR;
foreach $file ( @files ) {
if( $file =~ /log\..*/ ) {
system( "cp $ext_build/$file $pub_ext_log" );
if( $? >> 8 ) {
c_die("Can't cp $ext_build/$file $pub_ext_log\n");
}
}
}
}
exit 1;

sub c_die {
my( $msg ) = @_;
print $msg;
exit 3;
}

remote_post

[firebird.cs.wisc.edu:build] cat remote_post
#!/usr/bin/env perl

######################################################################
# $Id: remote_post,v 1.1.2.2 2004/06/25 06:32:23 wright Exp $
# post script to cleanup after a Condor build, successful or not
######################################################################

# autoflush our STDOUT
$| = 1;

# This is debugging output for the sake of the NWO infrastructure.
# However, it might be useful to us some day so we can see what's
# going on in case of failures...
if( defined $ENV{_NMI_STEP_FAILED} ) {
my $nmi_task_failure = "$ENV{_NMI_STEP_FAILED}";
print "The value of _NMI_STEP_FAILED is: '$nmi_task_failure'\n";
} else {
print "The _NMI_STEP_FAILED variable is not set\n";
}

######################################################################
# tar up test results
######################################################################

print "Tarring up results\n";
open( TAR, "tar zcf results.tar.gz public|" ) ||
die "Can't open tar as a pipe: $!\n";
while( <TAR> ) {
print;
}
close( TAR );
$tarstatus = $? >> 8;
if( $tarstatus ) {
die "Can't tar zcf results.tar.gz public: status $tarstatus\n";
}
print "Done tarring results\n";
exit 0;

post_all

[firebird.cs.wisc.edu:build] cat post_all
#!/usr/bin/env perl

######################################################################
# $Id: post_all,v 1.1.2.3 2004/06/29 05:10:40 wright Exp $
# Submit-side NMI build system infrastructure to save results after
# all the platform-specific jobs have completed
######################################################################

use Cwd;

# autoflush our STDOUT
$| = 1;

my $BaseDir = getcwd();
my $CommonDir = "$BaseDir/common";
my $vers_file = "CONDOR-VERSION";
my $result = "results.tar.gz";
my $src_file;

chdir( "$CommonDir" ) || die "Can't chdir($CommonDir): $!\n";

open( VERS, "$vers_file" ) || die "Can't open $vers_file: $!\n";
while( <VERS> ) {
$vers = $_;
}
close( VERS );
$src_file = "condor-$vers.tar.gz";
-f $src_file || die "$src_file does not exist!\n";

# All we want to do is tar the files in common into
# common/results.tar.gz so that we can save them.
print "Tarring up contents of common to save for later\n";
open( TAR, "tar -zcvf $result $vers_file $src_file|" ) ||
die "Can't open(tar -zcvf $result $vers_file $src_file): $!\n";
while( <TAR> ) {
print $_;
}
close( TAR );
if( $? ) {
die "Tar failed with status $?\n";
}
print "tar completed successfully\n";
print "Removing files:\n";
foreach $file ("$vers_file", "$src_file") {
unlink( $file ) || die "Can't remove $file: $!\n";
print " $file\n";
}

print "All steps completed successfully\n";
exit 0;
[firebird.cs.wisc.edu:build]