How to Write an Sbatch Job Script

Q: Can I use both –mem and –mem-per-cpu in the same script?

No — per the Slurm documentation, --mem , --mem-per-cpu , and --mem-per-gpu are mutually exclusive. Use --mem for a fixed total regardless of core count, --mem-per-cpu when memory needs scale with the number of cores requested.

An sbatch script is the standard way to submit a batch job to a Slurm-managed compute cluster: an ordinary shell script with a block of special comment lines — #SBATCH directives — that tell Slurm what resources to allocate before your program runs. Get the directive block right and the scheduler queues, allocates, and runs your job unattended; get it wrong and you either wait far longer than necessary or your job dies partway through on a resource limit you didn’t realize you’d set.

This guide covers the anatomy of a correct sbatch script, which directives matter for almost everyone versus which are cluster-specific, a complete worked example, and the distinction between requesting compute resources and your program actually using them — a gap that quietly wastes allocation on shared clusters more than almost any other single mistake.

The anatomy of an sbatch script

An sbatch script has three parts, in a fixed order: the shebang, the #SBATCH directive block, and the commands that actually run your job.

1. The shebang

The first line must be a shebang telling the system which shell interprets the script, almost always:

#!/bin/bash

bash is the near-universal choice; some clusters also support #!/bin/sh or #!/bin/tcsh, but check local documentation before assuming a non-bash shell is supported, since site-specific module systems and environment setup are frequently written assuming bash.

2. The #SBATCH directive block

Immediately after the shebang comes a block of lines starting with #SBATCH, one directive per line. Slurm’s own documentation is explicit about a rule that trips up a lot of new users: directive parsing stops at the first non-comment, non-blank line. Once the script reaches its first real command, no #SBATCH line after that point is read — it’s just a comment at that point, silently ignored. Practically, that means every directive belongs together at the top, before any module load, export, cd, or program invocation.

Directives use long-form --option=value or short-form -x value syntax:

#SBATCH --job-name=my_analysis
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --output=my_analysis_%j.out
#SBATCH --error=my_analysis_%j.err
#SBATCH --partition=general

3. The commands

After the directive block, the script runs like any other shell script: load software (typically via an environment module system), set variables, and invoke your program.

The directives that matter for most users

Slurm has dozens of sbatch options; most users only ever need a handful. Per the official Slurm sbatch documentation:

--job-name / -J — a label for the job as it appears in squeue and accounting records. Defaults to the script’s filename if omitted. Purely cosmetic, but a real name makes a queue with dozens of your own jobs much easier to read than a wall of identical script names.
--time / -t — the wall-clock time limit, after which Slurm kills the job regardless of whether it finished. Accepts several formats: minutes, minutes:seconds, hours:minutes:seconds, days-hours, days-hours:minutes, or days-hours:minutes:seconds — so --time=02:00:00 and --time=1-00:00:00 are both valid (two hours, one day). Omit it and the job inherits the partition’s default time limit, which varies by cluster and can be short. Get this wrong in either direction and it costs you: too low and a long job gets killed just before finishing; too high and your job may sit longer in queue, since schedulers often favor jobs fitting into shorter backfill windows.
--nodes / -N and --ntasks / -n — how many physical nodes and how many tasks (processes) to allocate. For a typical single-machine job, --nodes=1 plus either --ntasks=1 (serial) or --ntasks=N (an MPI job launching N ranks) covers most cases. sbatch only requests the allocation — it doesn’t launch your tasks; you (or srun inside the script) still start the actual processes.
--mem or --mem-per-cpu — memory to allocate, either as a total per node (--mem=16G) or per allocated CPU (--mem-per-cpu=4G). Mutually exclusive with each other and with --mem-per-gpu — use one, not both. If neither is set, the job gets a cluster-configured default (DefMemPerNode/DefMemPerCPU), which varies by site and can be low. Exceeding the memory you requested is one of the most common reasons a job is killed by the scheduler, often with an out-of-memory message in the Slurm log rather than your program’s own error output.
--output / -o and --error / -e — where stdout and stderr are written. Both support substitution patterns: %j expands to the job ID, %A/%a to the array job ID and array task index. If --error is omitted, stderr merges into the --output file. Without either set, Slurm defaults to slurm-%j.out, which works but is harder to keep organized across many jobs than a descriptive filename with %j in it.
--partition / -p — which queue (partition) the job is submitted to. Clusters typically define several — general-purpose, short-job/debug, GPU, high-memory — each with its own limits. Omitting this lets the controller assign a default partition, but on a shared cluster that default is frequently not the queue you actually want; check sinfo for the available partition names before submitting anything non-trivial.

Directives that are cluster-specific — check local documentation, don’t copy blindly

A common source of sbatch failures: copying a script from a colleague or tutorial and submitting it unchanged on a different cluster. Several directives depend entirely on local configuration, and a value correct on one system can be silently wrong — or outright rejected — on another:

Partition and QOS names (--partition, --qos) are defined per-cluster; there’s no standard naming across institutions. A script that says --partition=gpu fails on a cluster that calls its GPU queue gpu-a100 or has no GPU partition at all.
Account/allocation directives (--account, -A) tie the job to a specific grant for accounting. Many clusters require this and reject submissions without it; the value is assigned by your cluster’s allocation system, not something to guess.
GPU request syntax (--gres=gpu:1, or newer --gpus=1/--gpus-per-task forms) varies by Slurm version and site configuration. Some clusters also require a GPU type (e.g. --gres=gpu:a100:1) and reject a bare request.
Default time and memory limits when you omit --time/--mem are set per-partition by the administrator (DefaultTime, DefMemPerNode/DefMemPerCPU in slurm.conf) and vary widely — there is no one universal default. Check with scontrol show partition rather than assuming a value from a different cluster carries over.
Constraint/feature flags (--constraint) select nodes with hardware features (a CPU generation, a filesystem, an interconnect) that administrators choose to tag — the available feature names are entirely local.

The general rule: directives that describe the shape of your job (time, nodes, tasks, memory, output files) are portable in syntax, though the numeric limits around them are not. Directives that name a specific cluster resource (partitions, QOS, accounts, GPU types, node features) are never portable — check your cluster’s own documentation or run sinfo/scontrol show partition before submitting rather than assuming a value from a different system.

A complete worked example

The following script submits a single-node job requesting 4 CPU cores and 16 GB of memory for up to 2 hours, and demonstrates a few good habits worth adopting as defaults:

#!/bin/bash
#SBATCH --job-name=rnaseq_align
#SBATCH --time=02:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --output=rnaseq_align_%j.out
#SBATCH --error=rnaseq_align_%j.err
#SBATCH --partition=general
#SBATCH --mail-type=END,FAIL
#SBATCH [email protected]

# Stop on first error, and treat unset variables as an error
set -euo pipefail

# Load required software via the cluster's module system
module load samtools/1.19
module load hisat2/2.2.1

# Move to the working directory for this job
cd "$SLURM_SUBMIT_DIR"

echo "Job ${SLURM_JOB_ID} running on $(hostname), allocated ${SLURM_CPUS_PER_TASK} CPUs"

# Pass the allocated core count to the program explicitly —
# Slurm reserves the cores, it does not tell your program to use them
hisat2 -p "${SLURM_CPUS_PER_TASK}" -x genome_index 
    -1 sample_R1.fastq.gz -2 sample_R2.fastq.gz 
    -S aligned.sam

samtools sort -@ "${SLURM_CPUS_PER_TASK}" -o aligned.sorted.bam aligned.sam

A few things worth noting about this example:

set -euo pipefail is standard bash defensive practice, not a Slurm feature — the official sbatch documentation doesn’t specify exit-on-error behavior because that’s a property of the shell running your script, not of Slurm itself. Without it, a failed command partway through a multi-step script can be silently skipped, and the job will report success even though a critical step never ran.
$SLURM_JOB_ID, $SLURM_SUBMIT_DIR, and $SLURM_CPUS_PER_TASK are environment variables Slurm sets automatically inside the job — useful for logging, for returning to the directory you submitted from, and, critically, for telling your program how many cores it was actually allocated (see below).
The %j in the output/error filenames expands to the numeric job ID, so repeated submissions of this script don’t overwrite each other’s logs.

Requesting resources vs. your program actually using them

This is the gap that causes the most quiet waste on shared clusters. Slurm’s directive block only reserves resources at the scheduler level — it does not modify your program’s behavior. Setting --cpus-per-task=8 allocates eight cores to your job and blocks other users from those cores for the job’s duration; it does nothing to make your program actually use eight cores. Whether it does depends entirely on the program:

Single-threaded code (a plain Python or R script with no explicit parallelism, most single-process CLI tools run without a threading flag) uses exactly one core no matter how many you request. Request 8, use 1, and the other 7 sit idle for the whole job — billed to your allocation, unavailable to anyone else, doing nothing.
Multi-threaded tools (many bioinformatics aligners, compilers, some numerical libraries) typically need to be told explicitly how many threads to use, via a flag — commonly -p, -t, --threads — or an environment variable like OMP_NUM_THREADS. This is what the worked example above does by passing -p "${SLURM_CPUS_PER_TASK}" to hisat2 rather than hardcoding a number — the program matches whatever Slurm actually allocated, so the two can never drift apart if the directive changes later.
MPI programs are launched across allocated tasks via srun or mpirun inside the script, not by --ntasks alone; sbatch creates the allocation, but something inside the script still has to start the parallel processes.
GPU workloads have the same gap: requesting a GPU with --gres=gpu:1 makes it visible to the job, but a program that isn’t built or configured for GPU execution simply won’t use it — the allocation still burns queue priority and GPU-hours as though it had.

The practical check, especially before scaling a job up across many samples or array tasks: after a job finishes, look at its actual CPU/memory efficiency (many clusters expose this via seff ) rather than assuming the directive block tells the whole story. A job that requested 8 cores and 32 GB but used 1 core and 3 GB isn’t just wasteful for you — on a shared cluster it delays other users’ jobs by holding resources the scheduler could otherwise allocate to them.

Frequently asked questions

Do #SBATCH directives have to be at the very top of the script?

They have to come before the first non-comment, non-blank line. Extra blank lines or # comments among the directives are fine, but once the script reaches an actual command, Slurm stops reading #SBATCH lines entirely, even if more appear further down. Keep the whole block together, immediately after the shebang.

What happens if I don’t set –mem or –time?

The job gets the partition’s configured default for whichever one you omit. Those defaults are set per-cluster by the administrator and vary widely — check scontrol show partition rather than assuming. In practice, omitting --time is riskier than omitting --mem: a default that’s shorter than your job needs means Slurm kills it partway through with no automatic retry.

Can I use both –mem and –mem-per-cpu in the same script?

No — per the Slurm documentation, --mem, --mem-per-cpu, and --mem-per-gpu are mutually exclusive. Use --mem for a fixed total regardless of core count, --mem-per-cpu when memory needs scale with the number of cores requested.

Why does my job say it ran out of memory when my program never reported an error?

Slurm enforces the requested memory limit independently of your program’s own error handling. Exceed --mem/--mem-per-cpu and Slurm kills the process directly — the failure shows up in the Slurm output/error log, often as an out-of-memory or “Exceeded job memory limit” message, rather than as an exception your program raised, since it’s terminated before it gets the chance.

Does requesting more nodes or cores make my job run faster?

Only if the program is actually written or configured to use them — see the section above. A serial script sees no speedup from a larger allocation; a poorly-scaling parallel program may see diminishing returns past a certain core count. Requesting more than your program can use just adds queue wait time without adding throughput.