CS/ECE 752 Fall 2021

Note that as part of this script we included a tar command to tar the build directory. This is important because in CHTC if you don't, the build directory won't be sent back to the submit server (i.e., where you submitted the job from). This will also applies to m5out when running a job, which we'll discuss in Step 4. But, because of the need to tar build, you will also have to untar the build directory once the job completes (e.g., untar it back into the gem5 folder). Additionally, note that the above docker just contains the required software to build gem5. You will still need to clone gem5 into the gem5 folder before launching any jobs.

C: Finally, now that both scripts are created, we can launch our job (e.g., condor_submit build-gem5.sub if build-gem5.sub is the name of your submission script). What this will do is tell the CHTC system that you have a new job you'd like run and you're launching it from your submit server. Underneath the hood, CHTC will look across the available machines and find one that is (a) able to support your job and (b) currently has at least some spare cycles to use on your job. One of the cool things about CHTC is it actually may move your job as it's executing across multiple machines -- without your needing to be involved! Also we recommend making this shell script executable (e.g., chmod u+x build-gem5.sh).

D: After launching the job, you'll need to wait a little bit for CHTC to do its magic and compile the job. Upon lauching, you should see CHTC print some information to your console with the ID of the job you launched. To check and see if that job is still running (or held), you can use condor_q. Eventually, you should see that your job completed though, and is no longer on the list of running condor jobs (i.e., when you run condor_q, your gem5 job is no longer there. Once this is done, you can progress to step 3!

Sidenote: this process here is what CHTC refers to as a "regular job." Another option is to submit an "interactive job" where you get dedicated access to a machine and run the necessary commands to do whatever you want to do interactively. Since that is not necssary for gem5, and because there are many fewer interactive machines in the CHTC cluster, we use a regular job instead.

Step 3: Compiling DAXPY

A & B: Now that we've compiled gem5, the next thing we need to do is compile the application we want to run in gem5. Just like when we compiled gem5, we'll create a submission file and shell script for compiling DAXPY.

	      # build-daxpy.sub
	      # build DAXPY submission script
	      universe = docker
	      docker_image = gcr.io/gem5-test/ubuntu-20.04_all-dependencies

	      log = daxpy.log
	      error = daxpy.err
	      output = daxpy.out
	      
	      executable = build-daxpy.sh

	      should_transfer_files = YES
	      when_to_transfer_output = ON_EXIT
	      transfer_input_files = daxpy.cpp

	      environment = "TERM=xterm-256color"
	      
	      request_cpus = 1
	      request_memory = 1GB
	      request_disk = 1GB

	      stream_output = True

	      queue 1
	    

---

	      #!/bin/bash
	      # build-daxpy.sh
	      # build DAXPY shell script

	      g++ -std=c++14 daxpy.cpp -o daxpy
	    

Note that our submission script looks pretty similar to the one for building gem5. Except here we're pointing to the daxpy files we want to compile and the daxpy shell script. Similarly, the shell script for building DAXPY simply specifies how to compile DAXPY with g++.

C & D: Once you've created the above scripts, submit your "build DAXPY" job just like you did the gem5 build job, wait for it to complete (should be much faster than building gem5), and proceed to Step 4!

Step 4: Running DAXPY in gem5 on CHTC

Important: Please see FAQ for how to avoid clobbering your m5out directory -- by default CHTC assumes multiple jobs in parallel write to the same output folder.

A & B: Now that we've compiled both DAXPY and gem5, we can finally run DAXPY in gem5 on CHTC! Just like the last two steps, we need to create a submission and shell script for running DAXPY in gem5 on CHTC:

	      # run-daxpy.sub
	      # running DAXPY in gem5 submission file
	      universe = docker
	      docker_image = gcr.io/gem5-test/ubuntu-20.04_all-dependencies

	      executable = run-daxpy.sh
	      # if we want DAXPY to be size 1K
	      arguments = 1024

	      should_transfer_files = YES
	      when_to_transfer_output = ON_EXIT
	      transfer_input_files = gem5/, daxpy

	      environment = "TERM=xterm-256color"
	      
	      request_cpus = 1
	      request_memory = 8GB
	      request_disk = 12GB

	      stream_output = True

	      queue 1
	    

---

	      #!/bin/bash
	      # run-daxpy.sh
	      # running DAXPY in gem5 shell script

	      mkdir m5out

	      build/X86/gem5.opt -r configs/example/se.py --cpu-type=MinorCPU --ruby -c daxpy --options="$1"

	      tar -czf m5out.tar m5out
	    

Most importantly, note that for command line arguments (e.g., the size of DAXPY) we utilize the "options" field, and tell it to take the first argument passed in the command line -- so when we launch this job, we should also specify the size of DAXPY. We also updated the submission file accordingly to specify these arguments (e.g., above we specify 1024 as the input size for DAXPY with arguments. As in Step 2, note that here we need to tar the output (which means on the submit server you'll need to untar it after the job is done). Similarly, since we're only running a single-threaded job, we only need 1 CPU for this test (unlike in Step 2). Moreover, you'll need to update the config file, cpu-type, and the application as appropriate for the gem5 config you want to use, the job you want to run, and the CPU you want to test on, respectively.

C & D: Now that we've made our scripts, as in Steps 2-3, we need to submit the CHTC job, and wait for it to return. Crucially, note that you can launch all of the experiments you want to run in parallel!

Step 5: Parsing the output

Once your experiments are done, you should scp the m5outs from the submit server to some local machine (e.g., the CSL machines) and parse the outputs there. The two main reasons are: (a) the submit servers are mostly intended for launching jobs, not for post-processing data and (b) since CHTC is its own separate file system, it's useful/important to copy it elsewhere so you can reference it later.

FAQ

1. How do I get HTCondor to email me after my jobs complete/fail: See here.
2. What if my job has in the "Held" state?: Usually this means there was an error of some kind with what you are running. Try checking the err output (perhaps using some of the tricks below in the FAQ). In my case, I usually check the log output first -- e.g., to see if a file couldn't be found by HTCondor.
3. How do I avoid overwriting m5out for different runs from the same directory?: Since you may be running multiple configurations from the same application folder/directory, if you are not careful you will overwrite your previous runs m5out folder with your next runs m5 out folder! Obviously, this would be less than desireable. So, there are a few ways to workaround this:
   1. (simplest) Name the m5out directory uniquely per job you are launching. One option here is to append a timestamp for when the job is launched to the m5out directory. The traditional way to do this is to pass an argument from your submit file to your shell file that uniquely identifies the m5out directory:
      
      # submission file

      		      ...
      		      arguments = 1024 $(ClusterId)
      		      ...
      		    

      # shell file

      		      mkdir m5out.$2
      
      		      ...
      
      		      tar m5out.$2.tar m5out.$2
      		    

      
      So, if the job ID is 182530.0, then the resulting m5out directory would be m5out.182530/. Likewise, I also suggest you update the log, output, and error lines in your submit file to be unique per job, e.g.,: output = out.log.$(ClusterId), to make it easy to check what happened in your runs.
   2. Use different initialdirs in the submit file. This essentially requires that you have a different directory for each run you want to do, but does a good job of isolating things. It also means you should add the "initialdir" field to your submission scripts to specify where the job should be run from. If you go this route, keep in mind that transfer_input_files uses initialdir as the base directory for transfer_input_files, and not the directory you're in when running condor_submit.
   3. (most complex, least amount of sub/shell files) See the CHTC User Guide, especially Chapter 4, for additional options that include batching multiple jobs together in a single sub/shell script combo -- this would also allow you to set the appropriate initialdir/outputdir based on the arguments you use to express this information.
4. Is there a way to enable CHTC to show the partial output when my job is running?: Yes! There are three options:
   1. (Preferred in most cases - should work everywhere on CHTC) Run "condor_ssh_to_job $JOBID". This will start a SSH terminal inside the job's environment. You can not only look at the files, but poke at the executable if you want.
   2. (Works everywhere, even when condor_ssh_to_job doesn't) Run "condor_tail $JOBID _condor_stdout". Acts a bit like the Unix 'tail' utility.
   3. (Must be specified at submit-time) Set "stream_output = True" in the submit file.
5. Can I use more than 8 cores for more building gem5 job?: Yes, you can. However, the more CPUs you request, the longer your wait time will likely be. This is because the are substantially fewer 16 core machines in the cluster than there are 8 core machines. The CHTC suspects the wait time for a 16 core machine will be longer than it will take to compile gem5 in a 8 core machine, but your mileage may vary.