Next Generation Build System

Desired Features

The aim of the new world order build system is to separate the functionality of the build system frame work and the component related build scripts. This is achieved by having a clear semantics on how the component related scripts will be invoked.

It should have following features –

1. Easy of use.
2. Easy of adding new components without modifying the build system code.
3. Easy of adding new platforms preferable without modifying the build system code.
4. Building multiple components at the same time if they are independent and follow the dependencies when required.
5. Log reporting in xml format.
6. Log the build related information in the database.
7. Should transfer the required externals for a particular component to the remote build machine. This can even include externals like Java. Flexibility on this feature is debatable.
8. Build system should be smart enough to handle the operating system signals and should report failure if any of the scripts is killed.
9. Web interface for component builders to build over the web.

Semantics

The build system functionality is divided into two

1. Tasks done at the submit machine
2. Task done at the remote machine

These semantics are shown below. Lines written in "blue" are the functionalities provided by the build system. Lines written in "black" are the "Glue-Scripts" provided by the component builder (user). Glue-Scripts bind the two entities (build system scripts and component related scripts) together. Build system framework expects specific code of conduct from the Glue-Scripts but is not concerned with how they achieve their tasks. Lines in "green" are remarks or more information available to the user.

Submit-side Semantics

Submit side of the build system is mainly concerned with doing the preliminary tasks done before the component build is actually dispatched to remote machines and doing the post build tasks after the component build results arrive back from the remote machine.

User initiates the build by passing the component description file to the build system. This component description file has all the information that the build system requires to successfully build the component. A sample component description file is shown below. It contains the information about the component and where the glue scripts are and what glue script to invoke at a given time. Once this information is parsed by the build system it creates an initial database record for this run and gets a runid. It also creates a buildspace/runspace where the build related information is stored. It creates the required sub-directories based on the platform list. Once the stage is set build system fetches component sources as instructed in the component description file and places them in a known location. It then invokes the pre_all script for the component if declared. Once the general tasks are done build system creates condor submit file for the requested platforms and submits a condor job that runs on the remote machine. Build system also invokes the platform_pre and platform_post scripts if declared at appropriate time to complete tasks related to the component build.

Once the condor job completes (successfully or unsuccessfully) build system invokes the post_all script. It then updates the database with the result status of each task.

Note: database updates and retrieval not implemented yet.

Execute-side Semantics (per platform)

Once the condor job (job to build the component on a targeted platform) comes to the execute machine, the very first script to be invoked is the build system wrapper script. After this the user is given a chance to split the build into several tasks of his/her choice in the remote_declare. If the user needs to take any action before declaring the remote tasklist he/she can do it in remote_pre_declare. Similarly if there are any tasks to be executed before the first build task but after the remote_declare, this can be done in remote_pre.

Then for every task declared by the user the build system runs them saves the statistics and results for the task and updates the database. When the component is built and all the tasks are completed user can declare a remote_post script. This essentially serves for the tasks like packaging the component in a particular way which is not a part of the build but is required for component distribution. the build system framework expects that the glue scripts create a single tar file called "results.tar.gz" that contains everything that is to be transferred back to the submit side. Stdout and Stderr are transferred back by default an other user log files created should be included in results.tar.gz by the user.

Note: database updates and retrieval not implemented yet.

Description of parameters used in the component description file

Component Description File Examples

Examples below show the command description files for GsiOpenssh and condor. The scripts used to build the products are also linked from the description file.

Assuming that your description is in componenet.desc file to build the component run -

1. GsiOpenssh

2. Condor