Slurm Priority and Fairshare: Why Your Job Is Waiting in the Queue

A job that sits PENDING after sbatch returns a job ID has not failed — it is waiting its turn. Slurm’s default scheduling plugin, priority/multifactor, does not run jobs first-come-first-served. It scores every pending job on several independent factors and re-evaluates that score continuously, so two jobs submitted seconds apart can start hours apart, and a job submitted after yours can legitimately start before it. This guide covers how that score is built, why fairshare specifically penalizes recent heavy use rather than lifetime use, and how to read squeue and sprio to see exactly where your job stands and why.

The Multifactor Priority Plugin

Slurm computes a single composite priority number for every pending job from a weighted sum of factors. Per the official documentation, the formula is:

Job_priority =
    site_factor +
    (PriorityWeightAge)       * (age_factor) +
    (PriorityWeightAssoc)     * (assoc_factor) +
    (PriorityWeightFairshare) * (fair-share_factor) +
    (PriorityWeightJobSize)   * (job_size_factor) +
    (PriorityWeightPartition) * (partition_factor) +
    (PriorityWeightQOS)       * (QOS_factor) +
    SUM(TRES_weight_cpu * TRES_factor_cpu, ...)
    - nice_factor

Each factor (age_factor, fair-share_factor, and so on) is normalized to a value between 0.0 and 1.0, and each PriorityWeight* is an unsigned integer a site administrator sets in slurm.conf. The documented default weight for every factor is 0 — a factor contributes nothing to the score until a cluster administrator assigns it a weight, which is why the balance between fairshare and, say, job age varies a lot between clusters. A common example configuration weights fairshare an order of magnitude above age and job size (e.g. PriorityWeightFairshare=10000, PriorityWeightAge=1000, PriorityWeightJobSize=1000), which is why fairshare tends to dominate on clusters that enable it at all.

If your site hasn’t enabled multifactor priority, jobs schedule in submission order (FIFO) within each partition, modified only by backfill. Ask your cluster admin, or run sprio -w (below) — an empty or all-zero weight list means fairshare and age currently contribute nothing on your cluster.

Fairshare: Why Recent Usage Lowers Your Priority

Fairshare is the mechanism most people mean when they say a job is “losing priority” for using too much of the cluster. It compares an account’s or user’s actual resource consumption against the share of the cluster it has been allocated, and lowers priority for those running ahead of their allocation. Under Slurm’s Fair Tree algorithm (the current default fairshare calculation), each association’s standing is computed as a level fairshare ratio, LF = S / U, where S is its normalized share and U is its normalized usage. An association that has used less than its allocated share gets an LF above 1.0 (under-served, boosted); one that has used more gets an LF between 0.0 and 1.0 (over-served, penalized). Associations are then ranked against their siblings in the account tree by that ratio, and the rank converts into the 0.0-1.0 fairshare factor that feeds the priority formula above.

The key word is recent. Fairshare is explicitly self-correcting rather than a permanent penalty: usage is decayed over time under PriorityDecayHalfLife, so heavy consumption from weeks ago counts for less than heavy consumption from today, and eventually stops counting at all. Slurm’s documentation gives 7 days as the commonly configured half-life — a site can set it shorter for faster forgiveness or longer for a longer memory, or set it to 0 to disable decay entirely and track usage since the accounting database was created. Practically, this means a lab that burned a large allocation last month is not still being punished for it today; what matters is how the account’s usage compares to its share right now, weighted toward the recent past.

This is also why fairshare is an account/association-level mechanism, not a per-job one: it doesn’t ask “is this specific job big,” it asks “has the account submitting this job been consuming more than its fair share of the cluster lately.” Two identical jobs from two different accounts can get very different fairshare factors purely based on each account’s recent history.

The Other Factors: Age, Job Size, Partition, and QOS

Fairshare rarely acts alone. The other weighted factors in the formula above matter just as much for where a specific job actually lands:

• Age — the length of time a job has been eligible and waiting in the queue. This factor climbs the longer a job sits pending (up to a configurable cap, PriorityMaxAge), which is the built-in anti-starvation mechanism: even a job from a heavy-usage account eventually accumulates enough age priority to run.
• Job size — the number of nodes or CPUs requested, normalized against the cluster or partition size. Depending on site policy this factor can favor either larger jobs (to reward efficient use of the whole cluster) or smaller ones.
• Partition priority — each partition can carry its own priority weight, so a job submitted to a “high-priority” or dedicated partition (e.g. one restricted to a funded project’s own nodes) can outrank an otherwise-identical job in a shared partition.
• QOS (Quality of Service) — a QOS assigned to a job or association carries its own priority contribution, independent of fairshare, and QOS-level limits (job count, CPU-minutes, GPU count) can hold a job back regardless of how high its computed priority is.
• Nice value — a user can voluntarily lower their own job’s priority with sbatch --nice=<value>, which subtracts directly from the composite score. This is genuinely useful for a job you want to run last if the cluster is busy, without asking an admin to intervene.

Because every factor is weighted independently, a job can be “winning” on fairshare (a light-usage account) but still wait behind a heavily-aged job from a heavy-usage account, or vice versa — there is no single dominant rule without knowing your cluster’s configured weights.

Checking Where Your Job Actually Stands

Two commands answer “why is my job waiting,” and they answer different questions.

squeue shows current queue state, including the human-readable pending reason and the composite priority as a normalized number:

squeue -u $USER -o "%.18i %.9P %.8T %.10M %.6D %20R %Q"

The %R field prints the reason code (see the table below); %Q prints the job’s priority converted to a float between 0.0 and 1.0, which lets you compare it directly against other pending jobs’ priorities without decoding the raw integer.

sprio breaks that single number back down into the individual factors that produced it — this is the tool for “why is my priority what it is,” not just “what is it”:

# Full weighted breakdown for your own pending jobs
sprio -l -u $USER

# Just the site's configured weights, no job data
sprio -w

# Normalized (0.0-1.0) factors instead of weighted values
sprio -n -j <jobid>

sprio -l prints one column per factor — age, fairshare, job size, partition, and QOS by default, with nice and TRES factors available but not shown unless requested — so you can see, for example, that your fairshare column is near zero while your age column is climbing, which tells you the account’s recent usage is the limiting factor and time (not a resubmit) is what will fix it.

Reading squeue’s PENDING Reason Codes

The %R reason field is often more immediately useful than the priority number itself, because it tells you which category of problem you’re looking at:

Reason code	What it means
Resources	The job is otherwise eligible to run, but the nodes/CPUs/GPUs it needs are not currently free. This is the “healthy queue” reason — the cluster is simply busy.
Priority	One or more higher-priority jobs exist for this partition or reservation. Your job is eligible but is being out-scored, typically by fairshare or age, by other pending work.
Dependency	The job has an unsatisfied --dependency on another job (e.g. afterok:<jobid>) and will not be considered for scheduling until that dependency clears.
ReqNodeNotAvail	A node the job specifically required (via --nodelist or a feature constraint) is down, drained, or otherwise unavailable.
PartitionTimeLimit	The job’s requested walltime exceeds the partition’s current maximum time limit.
AssocGrp*Limit (e.g. AssocGrpCPUMinutesLimit)	The job’s association has hit an aggregate limit — total CPU-minutes, running jobs, submitted jobs, etc. — configured on that account.
QOSMax*Limit (e.g. QOSMaxJobsPerUserLimit)	The job would violate a per-job or per-user limit attached to its QOS, independent of the association-level limits above.
JobHeldUser / JobHeldAdmin	The job is held (via scontrol hold) by the submitting user or an administrator and will not be scheduled until released.
BeginTime	The job’s earliest allowed start time, set with --begin, hasn’t been reached yet.
InvalidAccount	The account named on the job (or the user’s default account) is invalid or not authorized on this cluster/partition.
BadConstraints	The job’s --constraint feature expression cannot be satisfied by any node in the partition — not a queueing problem, a request the cluster cannot fulfill as written.

Resources and Priority are the two you’ll see for a job that’s genuinely fine and just waiting; everything else in the table is closer to a configuration or request problem worth fixing on the job itself.

What You Can (and Can’t) Do About It

Fairshare and age are, by design, largely outside an individual submission’s control — they reflect account-level history and elapsed time, not anything about this specific job. What you can influence:

• Request only what you need. Over-requesting nodes, CPUs, memory, or walltime makes a job harder to backfill around and can push it toward the low end of the job-size factor on clusters that favor smaller requests. See the guide on writing an sbatch job script for getting resource requests right the first time.
• Split large sweeps into array jobs. A single enormous job waits for one big block of resources to free up all at once; the same workload as a Slurm job array can backfill into smaller gaps as they open, often starting substantially sooner. The same logic applies to genuinely embarrassingly parallel workloads that don’t need to run as one job at all.
• Pick the right submission command. Understanding the tradeoffs in srun vs. sbatch vs. salloc avoids holding an interactive allocation open (and accruing usage against your fairshare) while you’re not actually using it.
• Lower your own priority when it helps. --nice is the one lever a user has that directly subtracts from the priority score, useful for background or non-urgent work you don’t want competing with your own more urgent jobs.
• Ask about QOS and partition options. If your project has a dedicated allocation (for example an NSF ACCESS allocation rather than shared campus-recharge nodes), submitting to the correct partition/QOS for that allocation can matter more than anything about the job itself.

What won’t help: resubmitting the same job repeatedly. A new job ID gets a fresh age factor of zero and loses whatever age priority the original submission had already accumulated, which is usually a net loss.

Frequently Asked Questions

Why did a job submitted after mine start first?

Because Slurm schedules by composite priority, not submission order. The later job may belong to an account with a much higher fairshare factor (lighter recent usage), request a job size the cluster favors, or simply fit into a resource gap the scheduler’s backfill algorithm could use immediately while your job’s requested resources weren’t free yet.

Does using a lot of compute today hurt my priority tomorrow?

Yes, but only temporarily. Fairshare tracks decayed usage under PriorityDecayHalfLife (commonly 7 days), so recent heavy use lowers your fairshare factor now, but that effect fades on the same timescale rather than persisting indefinitely.

What’s the practical difference between the Resources and Priority reason codes?

Resources means nothing is currently free that matches your request — even the top-priority job would wait. Priority means resources may well be available, but other pending jobs are outscoring yours for them right now.

Can I see my account’s fairshare standing without pending jobs to check?

sshare -l (a separate but related command from the same accounting system) shows raw and effective fairshare values per account/user directly, independent of any specific job.

Why does sprio show nothing for my job?

sprio only reports on jobs still in the scheduling queue. Once a job transitions to RUNNING, its priority calculation is no longer relevant and it drops out of sprio‘s output; check squeue for its current state instead.

Does requesting a shorter walltime improve priority?

Not directly through the priority formula, but a shorter time limit gives the backfill scheduler more opportunities to slot your job into a gap that opens up between two other jobs’ reservations — which often gets a job running sooner even without a change in its formal priority score.