CS 368-4 (2011 Fall) — Day 9 Homework

Due Tuesday, November 29, at the start of class.

Goal

Run a job (or many!) on CHTC using Condor. This is our gateway into tapping the full potential of CHTC via scripting.

Tasks

This assignment is quite straightforward, but I will walk you through some of the set-up steps, just to be sure.

Using your CHTC account information, log on to submit-368.chtc.wisc.edu. You can ssh to the machine from any other machine, including your own desktop or laptop. If you are not sure how to ssh to a remote machine, check out the Remote Logins section of my Notes for Beginners page.
On the submit machine, make sure you can run basic commands like condor_status and condor_q. If those commands fail, contact me right away.

Copy the following Python script into a new file (perhaps named something like homework_09.py):

#!/usr/bin/env python

"""Homework for CS 368-4 (2011 Fall)
Assigned on Day 09, 2011-11-22
Written by Tim Cartwright
Submitted to CHTC by #YOUR_NAME#
"""

import getpass
import os
import platform
import socket
import sys
import time

input_filename = None
if len(sys.argv) > 1:
    input_filename = sys.argv[1]

print >> sys.stderr, __doc__
print 'Time    :', time.strftime('%Y-%m-%d (%a) %H:%M:%S %Z')
print 'Host    :', getpass.getuser(), '@', socket.gethostname()
uname = platform.uname()
print "System  :", uname[0], uname[2], uname[4]
print "Version :", platform.python_version()
print "Program :", sys.executable
print 'Script  :', os.path.abspath(__file__)
print

if input_filename is None:
    print 'No input filename given, all done!'
    sys.exit(0)

print 'Reading input file at', os.path.abspath(input_filename)
input_file_object = open(input_filename, 'r')
input_lines = input_file_object.readlines()
print 'Found %d lines in the input file' % (len(input_lines))
input_file_object.close()

output_file_object = open('homework-09-output.txt', 'w')
print >> output_file_object, 'First line of input file:'
print >> output_file_object, input_lines[0].rstrip('\n')
output_file_object.close()
print 'Wrote output file'

Edit the script above to contain your own name instead of the #YOUR_NAME# placeholder!
Use the shell to make the script executable by itself:
```
chmod 0755 homework_09.py
```
If you named the script something other than homework_09.py, substitute your actual script file name in command above instead.
Write a simple input text file of your own devising. Make sure it has a few lines of text.
Make sure that you can run your script successfully from the command line. First, without a command-line argument:
```
./homework_09.py
```
And then, with a single command-line argument, which is the name of your input text file:
```
./homework_09.py input-09.txt
```
If you cannot run your script like this, neither can Condor! Fix any problems with the script and/or your command until this step works perfectly.

When you provide a command-line argument, be sure to check out the output file created by the script: homework-09-output.txt.
OK, everything is ready for you now. Create your Condor submit file, and run your job! Try it without a command-line argument first, and once that works, try adding the command-line argument and input file. Get comfortable with all of the basic Condor commands and submit-file statements presented in the slides.
Be sure to look at all of the different files created by your job. There may be interesting details in there!

Condor Tips

Here are some things to observe and/or watch out for.

Your script will probably run fairly quickly, but not instantaneously. Keep an eye on the Condor queue and on your log file.
Watch your log file for messages like this:
```
007 (010.000.000) 11/13 06:26:43 Shadow exception!
```
The key phrase is the Shadow exception! part. If you see this message in your log file, probably repeated many times, remove your entire job cluster. Then, try running your script from the command-line again. Does it really work? If so, check over your submit file very carefully for typos. When this error happened to me (yes, I screwed up!), it was because I could not run my script from the command-line correctly.
All of the Condor commands have standard man pages. For example, to learn more about the condor_q command, type:
```
man condor_q
```
If you get really stuck, send me your submit file and most recent log file in an email, and I will try to help.
If you think something is broken with our submit machine or with the CHTC facility, let me know. But, before you do so, double-check all your own work carefully. There are quite a few people, including myself, who are monitoring the CHTC facility, so it is somewhat unlikely that there will be significant problems with it. Nonetheless, problems do crop up now and then…

Extra Challenges

Like writing code with Python itself, the more you play around with the basics of Condor and running jobs, the more you will learn and be ready for the next step(s). So, here are some things to try:

Set up and run prior homework assignments in CHTC. In particular, assignments #4, #5, #7, and #8 are quite suitable for this challenge. However, keep the following points in mind:
- You may need to modify the program to work without manual intervention (esp. #4)
- You will certainly need the #!/usr/bin/env python line at the top of each script
- Always run the chmod command (above) on your script file
- Always make sure you can run your script from the command-line before submitting
Do you have scripts from other classes or research projects that you want to run? If so, go ahead! All the caveats and tips above apply. And please restrict yourself to research and training activities here…
Feeling really adventuresome? Try running compiled code — that is, not a script. It is best to compile your code right on submit-368.chtc.wisc.edu, because the environment on that machine is most similar to the environment on (most of) the execute machines. Once again, be absolutely sure that your compiled program works from the command line on submit-368 before actually submitting it.

Reminders

Do the work yourself, consulting reasonable reference materials as needed; any reference material that gives you a complete or nearly complete solution to this problem or a similar one is not OK to use. Asking the instructors for help is OK, asking other students for help is not. All standard UW policies concerning student conduct (esp. UWS 14) and information technology apply to this course and assignment.

Hand In

A printout of:

Your submit file
Log file
(Standard) output file (from the output = line)
(Standard) error file (from the error = line)
Script output (i.e., made by the script)

all from a single run of the script. If possible, fit it all onto a single sheet of paper (double-sided, if need be). Be sure to put your own name in the initial comment block of the script. Identifying your work is important, or you may not receive appropriate credit.