qsub

OK, the preliminaries are out of the way and we are now ready to use the batch scheduler for its main purpose: submitting jobs to be run on the compute nodes. The command for this is qsub, and we will illustrate its use with the sim.R file from earlier.

There are a few important qsub flags to know: -cwd says to run in our current working directory; -q BIOSTAT specifies the queue; -j y joins stderr and stdout into a single log file; and -b y lets us pass a command directly rather than a shell script. So, let’s run the simulation on a compute node:

qsub -cwd -j y -q BIOSTAT -b y Rscript sim.R

After a little while, a file like sim-a3f9bx2k.rds will appear. You may also notice that a log file like Rscript.o1002669 appeared (your number will be different). This contains any output and errors from your submission. Typically it is not interesting (in this case it should be empty), but it is valuable when debugging or when your script prints important information.

To specify that the job requires multiple cores, or that it runs on specific cores, use the -pe and -l flags as discussed on the previous page.

qsub arrays

Often you’ll want to submit many independent runs at once – for example, many replications of the simulation from earlier. qsub supports this directly with what it calls array jobs: essentially, a loop over qsub calls, but handled internally by the scheduler rather than scripted by you. We can run

qsub -cwd -j y -q BIOSTAT -t 1-10 -b y Rscript sim.R

to submit 10 independent runs of sim.R, each saving its own sim-[hash].rds file. This is very convenient as it lets you change the number of jobs on the fly from the command line. After submitting, try monitoring the jobs with qstat (more on this on the next page).

In addition to being convenient, array jobs are also helpful for limiting overhead with respect to the grid scheduler. Submitting qsub many separate times creates more work for the batch scheduler than a single array job. This becomes more and more of a problem when a large number of jobs are submitted, so if at all possible, please submit your work as a single array job as opposed to a large number of separate jobs.

Managing queue usage with qsub arrays

When submitting large array jobs, it is quite easy to accidentally tie up all of a queue’s resources. Fortunately, there is an easy-to-use flag to control the maximum number of simultaneously executing nodes. For example, taking the previous qsub command, we can add the -tc flag to allow a maximum of two indices to execute at the same time:

qsub -cwd -j y -q BIOSTAT -tc 2 -t 1-10 -b y Rscript sim.R

Varying parameters across jobs

Sometimes you want each run to use different settings. For example, maybe each job analyzes a different subset of the data or uses different simulation settings. The number of the job inside a qsub array can be accessed via the SGE_TASK_ID environment variable. For example, to vary the sample size n from 4 to 20 in steps of 2 across a 10-task array, replace the line n <- 10 in sim.R with:

id <- as.integer(Sys.getenv("SGE_TASK_ID"))
n <- seq(4, 20, by = 2)[id]

The rest of sim.R is unchanged. Submit it just like any other array job:

qsub -cwd -j y -q BIOSTAT -t 1-10 -b y Rscript sim.R

Task 1 will use n = 4, task 2 will use n = 6, and so on up to task 10 with n = 20.

Quick reference

The qsub flags covered on this page and the previous one, gathered in one place: