Slurm –nodelist, –exclude, and –constraint: Targeting or Avoiding Compute Nodes

Three ways to control which nodes a Slurm job runs on

Slurm gives you three separate mechanisms for controlling node placement, and they solve different problems. --nodelist (short form -w) names specific nodes a job must run on. --exclude (short form -x) names specific nodes a job must avoid. --constraint (short form -C) asks the scheduler for nodes with a given feature — a CPU generation, a GPU type, an interconnect, a rack — without naming any node directly. All three are documented in Slurm’s sbatch and srun reference pages and work identically across sbatch, srun, and salloc.

The short version, before the detail: use --exclude to route around one or two known-bad nodes, and use --constraint for everything else. --nodelist is the one to reach for least often, because it hardcodes a node name into a script that keeps running long after the cluster’s node names have changed.

--nodelist / -w: require specific named nodes

--nodelist restricts the job to a specific, named set of nodes. Per the srun documentation, the argument is a comma-separated list of node names, a Slurm-style range expression, or a filename (the value is treated as a filename if it contains a / character):

#SBATCH --nodelist=node01,node02#SBATCH --nodelist=node[01-04]

Slurm sorts the supplied names regardless of the order you write them. If you combine --nodelist with --nodes/-N and the node list is larger than the requested node count, Slurm trims the allocation down to fit; if the node list can’t satisfy the requested count on its own, Slurm allocates additional nodes elsewhere to make up the difference. Either way, the named nodes are a floor, not necessarily the complete allocation — check what actually got allocated with squeue or scontrol show job rather than assuming the list is exhaustive.

This is the right tool when you genuinely need a specific physical machine — reproducing a bug tied to one node’s hardware, benchmarking a particular box, or working around a scheduler quirk during a live incident. It is the wrong tool for anything that will be re-run later: node names are an inventory detail, and inventory changes. A node gets decommissioned, a cluster gets re-racked, an institution merges two clusters under one scheduler — and a script with --nodelist=node07 baked into it either fails outright when that node no longer exists, or silently keeps targeting a node that no longer matches the reason it was picked in the first place.

--exclude / -x: avoid specific nodes

--exclude is the mirror image: it removes named nodes from consideration rather than requiring them. It takes the same argument format as --nodelist — a comma-separated list, a range expression, or a filename:

#SBATCH --exclude=node03#SBATCH --exclude=node[10-12],node15

This is the practical fix for the single most common ad hoc placement problem: one node in the partition is misbehaving — flaky GPU, degraded local scratch disk, a firmware issue the sysadmin hasn’t gotten to yet — and you need your job to land anywhere else while it gets fixed. --exclude is deliberately narrow and short-lived by design: it’s meant to route around a known, temporary problem, not to encode a permanent placement policy. If you find yourself maintaining a long-lived --exclude list across many job scripts, that’s a signal the cluster needs a real fix (draining the node, or tagging it with a feature that separates it from the rest) rather than dozens of scripts quietly working around it forever.

--constraint / -C: request nodes by feature, not by name

--constraint asks for nodes that have a particular feature rather than naming nodes at all. Features are administrator-defined labels attached to nodes in slurm.conf (or via a dynamic NodeFeatures plugin) — things like a CPU microarchitecture, a GPU model, an interconnect type, or a rack/location tag. A cluster with mixed hardware generations might label nodes skylake, icelake, or a100/v100, and a job simply asks for the property it needs:

#SBATCH --constraint=icelake#SBATCH --constraint=a100

Constraints support boolean operators for more precise requests, per the sbatch reference:

• AND (&) — every allocated node must have both features: --constraint="intel&gpu".
• OR (|) — a node needs at least one of the listed features, and different nodes in the same allocation can satisfy it differently: --constraint="intel|amd" can mix Intel and AMD nodes in one job.
• Matching OR ([...]) — like OR, but forces every node in the allocation to match the same branch, so you don’t get a mixed result: --constraint="[rack1|rack2|rack3|rack4]" puts the whole job on one rack, whichever it picks.
• Counts (*) — request an exact number of nodes per feature inside a matching-OR group: --constraint="[rack1*2&rack2*4]" asks for two nodes from rack1 and four from rack2.
• Parentheses — group sub-expressions to control evaluation order: --constraint="foo&(bar|baz)" requires foo plus at least one of bar or baz.

One caveat worth knowing before you rely on it: when srun is launched as a job step from inside an existing salloc or sbatch allocation, the constraint value is restricted to a single feature name — the AND/OR/matching-OR operators above are for the initial job allocation request, not for sub-steps inside one.

To see what features are actually defined on a cluster before writing a constraint, use sinfo with the features format specifier:

sinfo -o "%N %f"

which lists each node alongside its available (%f) features; %b shows features currently active, relevant on clusters using a dynamic NodeFeatures plugin where availability can change at runtime. sinfo -Nel gives a broader node-oriented view that includes a FEATURES column alongside state, CPU count, memory, and partition.

Why --constraint is usually the better default

The practical reason to prefer --constraint over --nodelist whenever you’re expressing a hardware requirement rather than routing around one specific known problem: a constraint describes what you need, and a node name describes where that happened to live when you wrote the script. Those are different kinds of information, and only one of them survives cluster changes.

• Node names churn; features don’t (if defined well). Nodes get decommissioned, renamed during a re-rack, or absorbed into a merged cluster with a different naming scheme. A script pinned to node047 breaks the moment that name stops resolving. A script asking for --constraint=a100 keeps working as long as some node still carries that label — including a brand-new one added after the script was written.
• Constraints scale across a growing fleet; nodelists don’t. If a cluster adds twelve more GPU nodes next quarter, every job already using --constraint=gpu can use them immediately with zero changes. A job hardcoded to a nodelist of the original GPU nodes never sees the new capacity unless someone manually edits every script that references it.
• Constraints are portable across job scripts and users. A shared cluster’s node-naming convention is usually undocumented tribal knowledge; a feature like skylake or infiniband is self-describing and reusable by anyone submitting to the same partition, which matters for reproducibility when a script is handed off or published alongside a paper.
• Constraints reduce scheduling friction. Naming three specific nodes forces the scheduler to wait for exactly those nodes to free up, even when ten functionally identical nodes are idle. A feature-based constraint lets the scheduler place the job on whichever matching node becomes available first, generally shortening queue wait.

--nodelist and --exclude both still have real, legitimate uses — reproducing a node-specific bug, isolating a known-bad machine, or debugging during a live incident are all cases where you genuinely need a specific node, not a category of node. The distinction to keep in mind is durability: --exclude is meant to be temporary (remove it once the node is fixed), --nodelist is meant to be exceptional, and --constraint is the one safe to leave in a script you expect to still be running in a year.

Combining the three

These flags aren’t mutually exclusive. A common real pattern is a feature constraint plus an exclusion for one node that’s currently flaky despite matching the feature:

#SBATCH --constraint=a100#SBATCH --exclude=gpu07

or a node list narrowed further by excluding one node within it:

#SBATCH --nodelist=node[01-08]#SBATCH --exclude=node05

--nodelist and --constraint can also combine, in which case Slurm must find nodes that satisfy both — the named nodes that also carry the requested feature. If no overlap exists, the job stays pending, so combine them only when you have a specific reason to believe the intersection is non-empty (e.g. you already know which of your named nodes have that GPU model installed).

For the broader mechanics of building a submission script around flags like these, see how to write an sbatch job script. If the reason you’re avoiding a node list is that you’re running many similar jobs and want the scheduler to place each one independently, Slurm job arrays and embarrassingly parallel job patterns cover the surrounding submission strategy. Clusters allocated through a national or regional program — see NSF ACCESS vs. campus recharge models and the Digital Research Alliance of Canada — are exactly where feature-based constraints matter most, since you rarely know or control the underlying node names on shared national infrastructure. Once a job lands, moving the resulting data off the cluster commonly goes through Globus.

FAQ

What’s the difference between --nodelist and --constraint?

--nodelist names specific nodes by their Slurm hostname; the job runs only on those exact machines. --constraint names a feature attached to potentially many nodes; the job runs on whichever matching node the scheduler assigns. Use --nodelist when you need one specific physical machine; use --constraint when you need a category of hardware and don’t care which specific node provides it.

Can I use --exclude and --constraint in the same job?

Yes. They compose: Slurm finds nodes matching the constraint, then removes any excluded nodes from that set before allocating. This is the normal pattern for “give me an A100 node, just not that one specific A100 node that’s currently misbehaving.”

How do I find out what constraint features exist on my cluster?

Run sinfo -o "%N %f" to list every node with its available features, or ask your HPC center’s documentation/support — feature names are set locally by the cluster’s administrators and aren’t standardized across institutions, so a100 on one cluster and gpu_a100 on another might both be valid but different labels for the same thing.

What happens if I request a constraint that no node satisfies?

The job stays in the queue in a pending state with a reason indicating the resource requirement can’t currently be met (this shows up in squeue‘s REASON column). It won’t fail outright — Slurm will keep waiting unless a time limit or dependency causes it to be cancelled — but if no node will ever have that feature, it waits forever, so it’s worth double-checking the feature name against sinfo -o "%N %f" before submitting.

Does --constraint work the same way in srun, sbatch, and salloc?

For the initial allocation request, yes — all three accept the full operator syntax (AND, OR, matching OR, counts). The one exception is an srun call made as a job step from inside an already-allocated salloc/sbatch session: there, the constraint value is limited to a single feature name, since the step is working within resources already allocated to the job rather than requesting a fresh allocation.