Snakemake

For simple jobs and arrays, qsub on its own is all you need. But once a job has multiple dependent steps, it’s worth using a workflow manager to keep track of what depends on what and to resubmit only what’s actually out of date. Snakemake is a popular choice, especially in R/Python-heavy analysis pipelines, and it works very well in HPC environments.

Installing Snakemake on the hpc

Snakemake is a Python package (hence the snake thing). There’s a module for it, but the version is extremely old, so I recommend installing your own copy with uv instead:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.13 ~/.venv
uv pip install --python ~/.venv/bin/python snakemake
uv pip install --python ~/.venv/bin/python snakemake-executor-plugin-cluster-generic

Then either run the following code or add it to your profile so that it’s always available:

source "$HOME/.venv/bin/activate"

The Snakefile

Snakemake uses a file called a Snakefile to describe how various outputs depend on other inputs. Here is a simple Snakefile using our sim.R and combine.R scripts from earlier.

container: "r-ver_latest.sif"

REPS = range(1, 11)

rule all:
    input: "results.rds"

rule sim:
    output: "sim-{rep}.rds"
    shell: "Rscript sim.R {output}"

rule combine:
    input: expand("sim-{rep}.rds", rep=REPS)
    output: "results.rds"
    shell: "Rscript combine.R {output} {input}"

The logic here is that Snakemake will start with the first rule (rule all), which depends on results.rds. results.rds is the output of rule combine, which depends on sim-1.rds, sim-2.rds, …, sim-10.rds. Finally, sim-1.rds is created by running Rscript sim.R sim-1.rds (this is described in rule sim).

The only slightly complicated part is the REPS = range(1, 11) at the top; this is just the Python version of REPS <- 1:10 in R. Once that variable is created, expand("sim-{rep}.rds", rep=REPS) expands out to a list of files sim-1.rds, sim-2.rds, …, sim-10.rds. You could manually type all this out, but using code to do it is obviously cleaner and more flexible.

Note that the Snakefile also specifies which container to use.

The Snakemake profile

But how do we tell Snakemake to submit jobs through qsub instead of running them locally? This configuration goes in a profile, which is a directory containing a file named config.yaml. Let’s create a directory called sge/ (for Sun Grid Engine) and put this config file there:

executor: cluster-generic
jobs: 50
cores: 50
latency-wait: 60
printshellcmds: true

software-deployment-method:
  - apptainer

cluster-generic-submit-cmd: "qsub -terse -cwd -j y -q BIOSTAT -pe smp {threads}"

-terse is required here (not just convenient): it makes qsub print just the job ID, which is how Snakemake tracks each job’s status.

jobs: 50 and cores: 50 specifies an upper limit on the number of cores and jobs running at once (like the -tc flag in qsub). You can override this from the command line, which we will do below; the above just sets up a default.

Note: If you end up using Snakemake across multiple projects, it’s worth putting this directory at ~/.config/snakemake/sge/ instead, so every project can reuse it with --profile sge without a copy sitting in each one.

Running snakemake

snakemake --profile sge --cores 2

What this code does:

  1. Submit sim.R through qsub 10 times (each one through our container), making sure that it never uses more than 2 cores at a time
  2. Once those jobs are all done running, qsub the combine.R script to combine the files into results.rds.

Meanwhile, setting the profile ensures that each job is run through qsub and apptainer. Note that the resulting results.rds object has 100,000 rows: 10,000 each from 10 different simulations.

If we left off --cores 2, all 10 sim.R jobs would run at once, but combine.R would still wait until all the sim.R jobs were finished before running.

This is not intended to be a full tutorial on Snakemake, but hopefully this example shows why the tool is useful and worth learning: it allows you to manage a complex chain of dependencies through simple rules describing the input and output of each step.