Slurm is an open-source job scheduler for Linux clusters. It decides which jobs run on which machines, in what order, and for how long. Submit your work to a queue, and Slurm finds the nodes, runs it, and tells you when it is done. It is used by more than half of the top 100 systems in the TOP500 list of supercomputers.
The name was originally an acronym for Simple Linux Utility for Resource Management. It is now just called the Slurm Workload Manager, and it is maintained by SchedMD, which NVIDIA acquired in December 2025. NVIDIA has committed to keeping it open source and vendor-neutral.
What Slurm actually does
A GPU cluster has a fixed amount of hardware and, usually, more people who want to use it than can be served at once. Without a scheduler you get the worst version of this: people messaging each other to ask whether node 4 is free, jobs colliding, and expensive hardware sitting idle overnight because nobody queued anything.
Slurm solves three problems at once:
- Allocation. It grants exclusive or shared access to specific nodes for a set period, so two jobs do not land on the same GPU.
- Execution. It starts your work on the nodes it allocated, monitors it, and cleans up afterwards.
- Arbitration. It holds pending work in a queue and decides what runs next when resources free up, according to policies you set.
That third one is the reason people adopt it. A scheduler is what turns a pile of machines into a shared resource with rules.
How Slurm is put together
Two daemons do the work.
slurmctld runs on a management node and is the brain: it holds the queue, makes scheduling decisions, and tracks the state of everything. In a Runpod Cluster this node is designated the Slurm Controller.
slurmd runs on every compute node, waits for instructions from the controller, launches the work, and reports back.
An optional third daemon, slurmdbd, records accounting data so you can see who used what.
Nodes are grouped into partitions, which are queues with their own rules. A common setup has a short partition with tight time limits for testing and a long one for real training runs, so a two-week job cannot block someone who needs twenty minutes.
The Slurm commands you will actually use
Six commands cover most day-to-day work.
salloc is worth knowing too: it reserves nodes and gives you a shell on them, which is the right tool when you want to poke at a problem interactively rather than submit a script.
The distinction between sbatch and srun trips up most newcomers. sbatch queues a script and returns immediately, so your job runs whenever resources free up and you can close your laptop. srun blocks, running the task in your current session. Use sbatch for anything that takes longer than a coffee break.
Prolog and epilog scripts
Slurm can run scripts before and after a job, which is how most clusters handle setup and cleanup without asking users to remember it.
A prolog runs before the job starts. Typical uses are checking node health, clearing GPU memory left by whatever ran previously, creating scratch directories, and refusing the job if the node looks wrong.
An epilog runs after the job finishes, whether it succeeded, failed or was cancelled. Typical uses are deleting scratch data, killing stray processes and resetting GPU state.
The practical reason these matter on GPU clusters: a job that crashes can leave a GPU in a bad state, and the next person to land on that node inherits the problem. An epilog that resets the device turns a mysterious recurring failure into a non-event. Both are configured by an administrator in slurm.conf, not by the person submitting the job.
Slurm vs Kubernetes
These get compared constantly, and they were built for different problems.
Kubernetes orchestrates long-running services. It assumes containers that should stay up, be replaced when they die, and scale with demand. It was designed in 2014, before GPUs were a first-class scheduling concern, and GPU support has been added since rather than designed in.
Slurm schedules finite jobs that end. It assumes work that is submitted, waits its turn, runs to completion and produces output. Batch training runs, simulations, sweeps. It has understood exclusive hardware allocation and multi-node coordination since long before either was fashionable.
Runpod does not support Kubernetes. We have written separately about why we take a different approach to GPU orchestration, and the short version is that the Kubernetes layer costs more than it returns for most GPU work.
When you need Slurm, and when you do not
You need a scheduler when you are sharing a cluster. More than a handful of people on shared hardware, and coordination stops being something you can do in chat. That is the threshold, not the size of the cluster.
You need one for multi-node training. Coordinating processes across nodes, assigning ranks, and cleaning up when one node fails is exactly what Slurm was built for. Doing it by hand is possible and unpleasant.
You need one for queued work. Hyperparameter sweeps, batch jobs overnight, anything where you want to submit a hundred runs and collect results later.
You do not need one for a single GPU. If you are one person on one machine, Slurm is overhead with no return. Run your script.
You do not need one for serving inference. A model behind an API is a service. Use something built for services, or a serverless platform that handles the scaling for you.
Running Slurm on Runpod
Clusters can come with Slurm pre-configured, with one node designated as the controller and the rest as compute nodes, so you are not installing and configuring it yourself. Clusters scale to 64 GPUs with 1600 to 3200 Gbps between nodes, which is the part that decides whether multi-node training actually scales. Our guide to NVLink, InfiniBand and Ethernet covers why that number matters more than the per-card specifications.
The Slurm client tooling ships in our official base image. slurm-wlm is installed in the image our PyTorch templates build on, so sbatch, srun, squeue, sinfo, scancel and sacct are on the path without installing anything. That means the commands are there, not that a scheduler is running: the controller and compute daemons still need a Cluster and the configuration below.
For the setup itself, including connecting to the controller node and running your first multi-node script, see setting up Slurm on Runpod Instant Clusters.
Billing is per second across Pods, Serverless and Clusters, with no ingress or egress fees. An H100 SXM is {{gpu:h100-sxm}}/hr on Secure Cloud and an H200 is {{gpu:h200}}/hr. Current rates are on the pricing page.
FAQ
What does Slurm stand for?
Slurm originally stood for Simple Linux Utility for Resource Management. The acronym has been dropped and it is now called the Slurm Workload Manager.
Is Slurm free?
Yes. Slurm is open source and free to use. Commercial support and development come from SchedMD, which NVIDIA acquired in December 2025 and has committed to keeping the software open source and vendor-neutral.
What is the difference between sbatch and srun?
sbatch submits a script to the queue and returns straight away, so the job runs when resources are available and you do not need to stay connected. srun runs a task immediately in your current session and blocks until it finishes. Use sbatch for real work and srun for quick interactive tasks or as a step inside a batch script.
Is Slurm better than Kubernetes?
Neither is better; they solve different problems. Slurm schedules jobs that run to completion, which is the shape of most training and simulation work. Kubernetes orchestrates services that stay running, which is the shape of an inference API. Choose based on whether your work finishes or persists.
What are Slurm prolog and epilog scripts?
A prolog runs before a job starts and an epilog runs after it ends. On GPU clusters they are typically used to check node health, clear GPU memory left behind by a previous job, create and delete scratch directories, and kill stray processes. Both are set by an administrator in the Slurm configuration rather than by the person submitting a job.
Do I need Slurm for a single GPU?
No. Slurm exists to arbitrate between competing demands on shared hardware. On one machine that you control, it adds configuration overhead and returns nothing. Run your script directly.
What is a Slurm partition?
A partition is a named queue of nodes with its own rules, such as maximum job length or who is allowed to submit. Clusters commonly run a short partition for testing and a longer one for production runs, so a multi-day job cannot block someone who needs a few minutes.
Get started
The quickest way to understand a scheduler is to submit a job to one. Runpod Clusters can come with Slurm already configured, and billing is per second with no minimum, so a first multi-node run costs very little. See current pricing or deploy a Cluster.
