Run parallel jobs on the compute grid

The Sun Grid Engine (SGE) is a standard interface to submitting jobs to a compute cluster. The department uses SGE to manage all sorts of spare computing resources, including some really high-powered machines that are only accessible through SGE. Using this interface can be a little bit hairy and bootstrapping yourself with the documentation is difficult, hence this HOWTO.

Decide on an Architecture

Before you start, you must decide on a machine architecture on which to run your job. Do you want to run on a 32-bit architecture or a 64-bit one? Well, there are pros and cons to each:

32-bit

Pro: no need to recompile your programs.

Con: there are only six 32-bit SGE hosts.

64-bit

Pro: there are sixty 64-bit SGE hosts, and of course you can cram way more data into main memory on a 64-bit architecture.

Con: every department machine that you can log into normally, from sixg to wheat to your own workstation, is 32-bit. Therefore all our programs are compiled on 32-bit machines. You must recompile your program and all the libraries it requires in order to run it on a 64-bit machine.

If you chose to use a 32-bit architecture, continue to the next section. If you chose 64-bit, keep reading...

Recompiling for a 64-bit Architecture

... I'll let you know when I figure it out...

Run Your Job

Write a shell script interface to your executable

To parallel jobs, SGE spawns a different instance of the job on every machine available, each time setting the environment variable $SGE_TASK_ID to a different value (see below for how to specify the range of values). The easiest way to run your executable is via a shell script that captures this and transforms it into a command-line argument for the program that's doing the real work. Here's an example script:

#!/usr/bin/tcsh
# 
# sge.sh
# 
# An example SGE array script

# SGE requires task IDs to start at 1, but the program expects 0
@ MACHINEINDEX=( $SGE_TASK_ID - 1 )

./obj/myProgram a_bunch_of default arguments $MACHINEINDEX

Submit the job to SGE

1. ssh sge --- Log into the machine sge
2. cd to the directory from which you want to run your shell script
3. Submit the following command, replacing <STUFF_IN_BRACKETS> with the appropriate arguments:

   qsub -l arch=<ARCH> -q <QUEUE> -m <NOTIFY> -t <RANGE> -wd <PATH> <SHELLSCRIPT>

   Arguments:

   • -l arch=<ARCH> --- your choice of either lx24-x86 to use 32-bit machines or lx24-amd64 to use 64-bit machines.
   • -q <QUEUE> --- optional. If you'd like, you can select a specific job queue to which to add your job. Choices include short.q, long.q, idle.q, and highmem.q. See tstaff's descriptions for more details.
   • -m <NOTIFY> --- optional. By default, SGE doesn't push much information about the state of your job to you. If you'd like to get notifications by email, pick any combination of the following options:
     • b --- get an email when your job begins
     • e --- get an email when your job ends
     • a --- get an email if your job is aborted
     • s --- get an email if your job is suspended
     • n --- don't get any emails. Monitor your job with qmon instead.
   • -t <RANGE> --- specify a range for the task IDs that will be sent to your parallel jobs. The range looks like this: MIN-MAX:STEP. This is a range specification similar to Matlab's; in particular, note that task IDs are one-indexed and the range is inclusive of its maximum value.
     • MIN is an integer ≥ 1 that represents the lowest ID that will be assigned.
     • MAX is an integer ≥ MIN that represents the highest ID that will be assigned.
     • STEP is an integer ≥ 1 that represents the step size taken between assigned task IDs.
   • -wd <PATH> --- optional. Set the working directory for the host executing your job to <PATH>.
   • <SHELLSCRIPT> --- the shell script to be executed by the SGE hosts.

This command just prints out a single string, indicating a problem if there is one, otherwise the job ID of the submitted job. SGE dumps standard out and standard error from all the instances of your shell script that get run on the various hosts to files named according to the job and task ID, in the directory where qsub was called.

See Also

• tstaff's documentation for our particular SGE installation
• Sun's own SGE documentation, in particular:
  • Submitting a simple job
  • Submitting batch jobs
  • Submitting an array job
• Hogwash, David McClosky's even simpler interface layer over SGE. Just write all the command you want run on separate lines of a text file, submit it, and you're done. There's a Python interface for scripting, too.