Working with data in files

So far, we've done really simple examples where the entire input to the program is just on the command-line. What do you do if you have data files to deal with? Let's walk through a short example.

First, let's make a program, call it analyze.sh that analyzes a text file that it is provided on the command-line.

#!/bin/sh

if [ $# -ne 1 ]; then
    echo "Usage: analyze.sh "
    exit 1
fi

echo "About to do a deep analysis of $1..."
echo "First, we convert it to all upper case (see $1.upper)"
tr "[:lower:]" "[:upper:]" < $1 > $1.upper

echo "Next, we find the 10 most common words (see $1.10)"
cat gettysburg | tr "[:upper:]" "[:lower:]" | tr -cs "[:alpha:]" "\n" | sort | uniq -c | sort --key=1,7 -n -r  | head -10 > $1.10

sleep 5

You also need a file to analyze. Put the following text into a file called gettysburg.

Four score and seven years ago our fathers brought forth on this
continent, a new nation, conceived in Liberty, and dedicated to the
proposition that all men are created equal.

Now we are engaged in a great civil war, testing whether that nation,
or any nation so conceived and so dedicated, can long endure. We are
met on a great battle-field of that war. We have come to dedicate a
portion of that field, as a final resting place for those who here
gave their lives that that nation might live. It is altogether fitting
and proper that we should do this.

But, in a larger sense, we can not dedicate -- we can not consecrate
-- we can not hallow -- this ground. The brave men, living and dead,
who struggled here, have consecrated it, far above our poor power to
add or detract. The world will little note, nor long remember what we
say here, but it can never forget what they did here. It is for us the
living, rather, to be dedicated here to the unfinished work which they
who fought here have thus far so nobly advanced. It is rather for us
to be here dedicated to the great task remaining before us -- that
from these honored dead we take increased devotion to that cause for
which they gave the last full measure of devotion -- that we here
highly resolve that these dead shall not have died in vain -- that
this nation, under God, shall have a new birth of freedom -- and that
government of the people, by the people, for the people, shall not
perish from the earth.

Our submit file looks nearly identical to what we had before, except for the one highlighted line that specifies the data file to transfer:

Universe        = vanilla
Executable      = analyze.sh
Output          = analyze.out
Error           = analyze.error
Log             = analyze.log
Arguments       = gettysburg
ShouldTransferFiles = Yes
WhenToTransferOutput = ON_EXIT
transfer_input_files = gettysburg
queue 

Notice that you just had to specify the input files and not the output files. Condor will automatically transfer back any new files, so you don't have to worry about it. Nifty, huh?

Now run the job.

% condor_submit submit
Submitting job(s).
Logging submit event(s).
1 job(s) submitted to cluster 60293.

% ls -lh gettys*
-rw-rw-r-- 1 zmiller zmiller 1.5K Jul  8 18:45 gettysburg
-rw-r--r-- 1 zmiller zmiller  120 Jul  8 18:59 gettysburg.10
-rw-r--r-- 1 zmiller zmiller 1.5K Jul  8 18:59 gettysburg.upper

You got your files! Check them out--do they look okay?

On your own

• washington-inaugural-address
• washington-insurrection-speech
• bush-inaugural-address
• obama-inaugural-address