Submitting your first Condor job

Objective of this exercise

The object of this exercise to have you run and understand your first Condor job, as well as run small sets of jobs in a parameter sweep. This is an important exercise because it is the basis for everything that follows. If there is anything you don't understand in this exercise, please ask before you continue on.

Because this is an important foundation, please seriously consider doing the “On Your Own” section.

First you need a job

Before you can submit a job to Condor, you need a job. We will quickly write a small program in C. If you aren't an expert C programmer, fear not. We will hold your hand throughout this process.

First, create a file called simple.c using your favorite editor. Put it anywhere you like in your home directory. In that file, put the following text. Copy and paste is a good choice: 

% mkdir -p ~/condor-test 
% cd ~/condor-test

% cat > simple.c
#include <stdio.h>

main(int argc, char **argv)
{
    int sleep_time;
    int input;
    int failure;

    if (argc != 3) {
        printf("Usage: simple <sleep-time> <integer>\n");
        failure = 1;
    } else {
        sleep_time = atoi(argv[1]);
        input      = atoi(argv[2]);

        printf("Thinking really hard for %d seconds...\n", sleep_time);
        sleep(sleep_time);
        printf("We calculated: %d\n", input * 2);
        failure = 0;
    }
    return failure;
}

type control-d here

Now compile that program: 

% gcc -o simple simple.c

% ls -lh simple
-rwxr-xr-x 1 zmiller guest 8,5K 2010-12-07 05:32 simple

Finally, run the program and tell it to sleep for four seconds and calculate 10 * 2: 

% ./simple 4 10
Thinking really hard for 4 seconds...
We calculated: 20

Great! You have a job you can tell Condor to run! Although it clearly isn't an interesting job, it models some of the aspects of a real scientific program: it takes a while to run and it does a calculation.

Think back to the lecture. I said that our first step was to have a job to run. Now we'll work on running it in Condor, and eventually running lots of copies of it.

Submitting your job

Now that you have a job, you just have to tell Condor to run it. Put the following text into a file called submit: 

Universe   = vanilla
Executable = simple
Arguments  = 4 10
Log        = simple.log
Output     = simple.out
Error      = simple.error
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT
Queue

Let's examine each of these lines:

Next, tell Condor to run your job: 

$ condor_submit submit 
Submitting job(s).
Logging submit event(s).
1 job(s) submitted to cluster 10.

Now, watch your job run: 

% condor_q
-- Submitter: treinamento01.ncc.unesp.br : <200.145.46.65:56001> : treinamento01.ncc.unesp.br
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
10.0   zmiller        12/7  05:33   0+00:00:03 R  0   0.0  simple 4 10        That R means it's running

1 jobs; 0 idle, 1 running, 0 held

% condor_q
-- Submitter: treinamento01.ncc.unesp.br : <200.145.46.65:56001> : treinamento01.ncc.unesp.br
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD                

0 jobs; 0 idle, 0 running, 0 held

Tip: if you see lots of jobs running that aren't yours (because your classmates are busy), jump ahead to "A few tips and tricks" to learn how to look at just your jobs and no others.

When my job was done, it was no longer listed. Because I told Condor to log information about my job, I can see what happened: 

% cat simple.log

000 (010.000.000) 12/07 05:33:52 Job submitted from host: <200.145.46.65:56001>
...
001 (010.000.000) 12/07 05:33:54 Job executing on host: <200.145.46.74:49619>
...
005 (010.000.000) 12/07 05:33:58 Job terminated.
        (1) Normal termination (return value 0)
                Usr 0 00:00:00, Sys 0 00:00:00  -  Run Remote Usage
                Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
                Usr 0 00:00:00, Sys 0 00:00:00  -  Total Remote Usage
                Usr 0 00:00:00, Sys 0 00:00:00  -  Total Local Usage
        56  -  Run Bytes Sent By Job
        8619  -  Run Bytes Received By Job
        56  -  Total Bytes Sent By Job
        8619  -  Total Bytes Received By Job
...

That looks good: the job started up quickly, though you will often see slightly slower startups. Condor doesn't optimize for fast job startup, but for high throughput, The job ran for about four seconds. 

% cat simple.out
Thinking really hard for 4 seconds...
We calculated: 20

Excellent! We ran our sophisticated scientific job on a Condor pool! We've only run one job though. Can we run more?

Doing a parameter sweep

If you only ever had to run a single job, you probably wouldn't need Condor. But we would like to have our program calculate a whole set of values for different inputs. How can we do that? Let's change our submit file to look like this: 

Universe   = vanilla
Executable = simple
Arguments  = 4 10
Log        = simple.$(Process).log
Output     = simple.$(Process).out
Error      = simple.$(Process).error
should_transfer_files   = YES
when_to_transfer_output = ON_EXIT
Queue

Arguments = 4 11
Queue

Arguments = 4 12
Queue

There are two important differences to notice here. First, the Log, Output and Error lines have the $(Process) macro in them. This means that the output and error files will be named according to the process number of the job. You'll see what this looks like in a moment. Second, we told Condor to run the same job an extra two times by adding extra Arguments and Queue statements. We are doing a parameter sweep on the values 10, 11, and 12. Let's see what happens: 

% condor_submit submit
Submitting job(s)...
Logging submit event(s)...
3 job(s) submitted to cluster 12.

% condor_q
-- Submitter: treinamento01.ncc.unesp.br : <200.145.46.65:56001> : treinamento01.ncc.unesp.br
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
  12.0   zmiller        12/7  05:38   0+00:00:00 I  0   0.0  simple 4 10       
  12.1   zmiller        12/7  05:38   0+00:00:00 I  0   0.0  simple 4 11       
  12.2   zmiller        12/7  05:38   0+00:00:00 I  0   0.0  simple 4 12       

3 jobs; 3 idle, 0 running, 0 held


% condor_q
-- Submitter: treinamento01.ncc.unesp.br : <200.145.46.65:56001> : treinamento01.ncc.unesp.br
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               
  12.0   zmiller        12/7  05:38   0+00:00:02 R  0   0.0  simple 4 10       
  12.1   zmiller        12/7  05:38   0+00:00:02 R  0   0.0  simple 4 11       
  12.2   zmiller        12/7  05:38   0+00:00:02 R  0   0.0  simple 4 12       

3 jobs; 0 idle, 3 running, 0 held

% condor_q
-- Submitter: vdt-itb.cs.wisc.edu : <198.51.254.90:39927> : vdt-itb.cs.wisc.edu
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD               

0 jobs; 0 idle, 0 running, 0 held

% ls simple*out
simple.0.out  simple.1.out  simple.2.out  simple.out

% cat simple.0.out
Thinking really hard for 4 seconds...
We calculated: 20

% cat simple.1.out
Thinking really hard for 4 seconds...
We calculated: 22

% cat simple.2.out
Thinking really hard for 4 seconds...
We calculated: 24

Notice that we had three jobs with the same cluster number, but different process numbers. They have the same cluster number because they were all submitted from the same submit file. When the jobs ran, they created three different output files, each with the desired output.

You are now ready to submit lots of jobs! Although this example was simple, Condor has many, many options so you can get a wide variety of behaviors. You can find many of these if you look at the documentation for condor_submit.

On your own

Now that you've gotten your feet wet, try a few things on your own.

Just one log file

There's no reason to have a separate log file for each job. Change your submit file so that it uses a single log file. Does it all still work?

New outputs for each run

You might have noticed that the output files were over-written when you re-ran the jobs. (That is, simple.1.out was just re-written.) That was okay for a simple exercise, but it might be very bad if you had wanted to keep around the results. Maybe you changed a parameter or rebuilt your program, and you want to compare the outputs.

Just like you used $(Process), you can also use $(Cluster). This will be a number from your job ID. For example, it would be 12 from the above example. Change your submit file to use $(Cluster) and $(Process). If you do two job submissions, will you have separate output files?

Lots of jobs

Instead of specifying the Arguments multiple times with multiple queue statements, try this: 

Arguments = $(Process) $(Cluster)
queue 50

What does it mean? What happens? Does it work as you expect?

(An aside: you might wish to be able to do math, something like $(Process)+1. Unfortunately, you can't do that.)

Challenges

If you have time and feel comfortable with the technical background, try these extra challenges. You'll need to peruse the Condor manual (particularly the manual page for condor_submit) to find answers. Feel free to ask Zach too--he'd love to give you hints!