qsub

We have now come to the heart of using an HPC cluster: instead of running your code yourself, interactively, you hand it to the scheduler and let it run on a compute node. Once it’s submitted, you’re free — no need to stay logged in or keep anything open while it runs. qsub is the command that does this, and we’ll illustrate it with the sim.R file from earlier:

qsub -cwd -j y -q BIOSTAT -b y Rscript sim.R

Briefly: -cwd runs the job in the current directory, -q BIOSTAT picks the queue, -j y combines stdout and stderr into one log file, and -b y lets us pass a command directly instead of a script file. (A full list of important flags is at the bottom of this page.)

After a little while, a file like sim-a3f9bx2k.rds will appear, along with a log file like Rscript.o1002669 (your number will be different) containing any output or errors — typically uninteresting, but valuable for debugging or if your script is intended to print something useful. In our case, it shows the R version, since sim.R prints it: [1] "R version 4.2.2 (2022-10-31)".

And that’s the whole workflow: you can close your laptop, log out, go to sleep, and come back the next day to find your job finished (or still running, if it’s a big one) — no persistent connection required.

qsub arrays

Often you’ll want to submit many independent runs at once — for example, many replications of the simulation from earlier. qsub supports this directly with what it calls array jobs: essentially, a loop over qsub calls, but without you having to write a script. We can run

qsub -cwd -j y -q BIOSTAT -t 1-10 -b y Rscript sim.R

to submit 10 independent runs of sim.R, each saving its own sim-[hash].rds file.

In addition to being convenient, array jobs are also helpful for limiting overhead with respect to the grid scheduler. Submitting qsub many separate times creates more work for the batch scheduler than a single array job. This becomes more and more of a problem when a large number of jobs are submitted, so if at all possible, please submit your work as a single array job as opposed to a large number of separate jobs.

Managing queue usage with qsub arrays

Submitting a large number of jobs is good — it means you’re getting the most use out of the HPC! However, you want to avoid tying up all of the queue’s resources. Fortunately, there is a flag to control the maximum number of simultaneous jobs. For example, taking the previous qsub command, we can add the -tc flag to cap the number of simulations executing at the same time (here, it’s limited to 2):

qsub -cwd -j y -q BIOSTAT -tc 2 -t 1-10 -b y Rscript sim.R

In this specific example, there’s nothing wrong with 10 jobs running at once, but if you submit, say, 200 jobs, you might want to add -tc 40 or something to make sure you’re not running more than 40 at any one time.

Varying parameters across jobs

Sometimes you want each run to use different settings. For example, maybe each job analyzes a different subset of the data or uses different simulation settings. The number of the job inside a qsub array can be accessed via the SGE_TASK_ID environment variable. For example, to vary the sample size n from 4 to 20 in steps of 2 across a 10-task array, replace the line n <- 10 in sim.R with:

id <- as.integer(Sys.getenv("SGE_TASK_ID"))
n <- seq(4, 20, by = 2)[id]

The rest of sim.R is unchanged. Submit it just like any other array job:

qsub -cwd -j y -q BIOSTAT -t 1-10 -b y Rscript sim.R

Task 1 will use n = 4, task 2 will use n = 6, and so on up to task 10 with n = 20.

Save yourself some typing

You may have noticed that we’ve typed -cwd -j y -b y on every qsub command. Unlike -q, -t, -pe, which you want to tailor to the job, these three flags you always want turned on. So you may want to set up an alias like this:

alias qs='qsub -cwd -j y -b y'

The single-job example from the top of this page then becomes:

qs -q BIOSTAT Rscript sim.R

I won’t use it in this tutorial just to keep everything explicit, but I do use it personally when I run things on the HPC.

Quick reference

The qsub flags covered on this page and the previous one, gathered in one place: