qbatch

Another option for easily submitting array jobs is qbatch. To use qbatch, create a text file with one command per line. For example:

R CMD BATCH --no-save --no-restore "--args 1" sim.R .sim.Rout
R CMD BATCH --no-save --no-restore "--args 2" sim.R .sim.Rout
R CMD BATCH --no-save --no-restore "--args hello" sim.R .sim.Rout
R CMD BATCH --no-save --no-restore "--args world" sim.R .sim.Rout

Then we can submit this as an array job with 4 cores (one per line) via:

qbatch -q BIOSTAT batch.txt

Note that doing so creates some temporary files in a hidden folder .qbatch and stores the batch.txt.o5508677.4, etc., files in a folder called logs (obviously, you can change these defaults if you want to).

If you want to have each job run more than one line, you can change the “chunk size”:

qbatch -q BIOSTAT -c 2 batch.txt

This will submit 2 jobs: job 1 will run lines 1 and 2, job 2 will run lines 3 and 4.

If you want each job to use more than 1 processor, you can change the processors per job:

qbatch -q BIOSTAT -c 2 --ppj 2 batch.txt

This will again submit 2 jobs, but run those jobs on 4 cores – each job is working with 2 cores. Note that these jobs still run consecutively, however (line 2 waits for line 1 to finish). If you want to run multiple commands in parallel, specify the -j argument:

qbatch -q BIOSTAT -c 2 --ppj 2 -j 2 batch.txt

This submit 2 jobs on 4 cores and engages all 4 cores simultaneously – job 1 runs lines 1 and 2 on two different processors, while job 2 runs lines 3 and 4 on different processors. This should provide all the flexibility you need to configure the submission of your jobs how you like, but consult the documentation for a full explanation of all available options.

Submitting a single shell command

Another thing qbatch is useful for is submission of single shell commands. For example:

qbatch -q BIOSTAT -- ls

This submits the simple listing command ls as a qsub job.

Most shell commands (such as ls) obviously don’t need to be submitted to compute nodes. However, this feature is often useful – for example, maybe you have an executable shell command fit_model that fits some time-consuming model. Perhaps you originally worked on it and run it interactively in a qlogin session, but now you need to refit the model for some reason. You could qlogin again, but it’s much more convenient to simply run qbatch -- fit_model.