How to stress test a GPU

To stress test a GPU, run a sustained full-load workload on it while watching temperature, clock speed, power draw and error counters. A healthy card holds its clocks under load and reports no errors. A failing one throttles hard, produces wrong results, or crashes. Fifteen minutes is usually enough to tell the difference.

Why you would stress test a rented GPU

The usual reason to stress test hardware is that you just bought it. Renting changes the question. You are not checking whether the card you own is faulty, you are checking whether the specific machine you have just been allocated is healthy before you commit hours of work to it.

That matters because failures on GPU infrastructure are rarely dramatic. A card with degraded memory does not refuse to start; it produces a NaN forty minutes into training. A node with a thermal problem does not crash; it quietly throttles and your job takes twice as long for the same money. A bad interconnect does not error; it just makes your eight-GPU run scale like four.

Ten minutes of checking at the start catches all three, and on per-second billing it costs a few cents.

What you are actually testing

Stress test covers five different things, and a tool that exercises one will happily pass a card that is broken in another.

• Compute. Sustained arithmetic at full occupancy. Catches clock instability and, at the extreme, silent calculation errors.
• Memory. Writing and reading patterns across the whole of VRAM. Catches the bad-memory faults that produce corrupted results rather than crashes.
• Thermals and power. Whether the card can hold its boost clocks, or drops them to stay within its temperature and power limits.
• Interconnect. On multi-GPU work, whether the links between cards deliver the bandwidth they should.
• The host. CPU, system RAM and storage, which is where a surprising share of the GPU is slow turns out to live.

The tools worth knowing

What you get out of the box on a Runpod pod. Our official base image, which the PyTorch templates build on, gives you nvidia-smi immediately. It also ships a compiler toolchain, so building gpu-burn is a git clone and a make with nothing to install first. DCGM and stress-ng are not preinstalled and need an apt-get install. Community templates set their own contents, so check rather than assume.

Reach for dcgmi diag when you want thoroughness. It checks things the others do not, including PCIe, power and NVLink. dcgmi diag -r 1 is a quick sanity check; -r 3 is a fuller run; -r 4 adds the memory test and takes considerably longer.

A ten-minute check before a long run

1. Look before you load. Run nvidia-smi and confirm the card is what you expected, the driver version is sane, and the memory is essentially free. If something else is already using several gigabytes, find out what.
2. Start monitoring in a second terminal. nvidia-smi dmon gives you a rolling view of clocks, temperature, power and utilization, which is what you will actually read the result from.
3. Load it. Run your chosen tool for at least five minutes. Shorter than that and you have tested the card’s ability to sprint, not its ability to hold a load once it warms up.
4. Watch what the clocks do. This is the whole test. A healthy card climbs to its boost clock and stays near it. Some drop under sustained load is normal and expected. A card that falls sharply and stays low is thermally limited or power limited.
5. Check the error counters. nvidia-smi -q reports ECC errors and retired pages on cards that support it. Correctable errors happen. Uncorrectable ones, or a page-retirement count that climbs while you watch, mean stop.
6. Run a short version of your real workload. A few training steps at your real batch size tells you more about whether this machine suits your job than any synthetic test does.

Testing CPU and GPU together

Most stress-testing advice treats the GPU in isolation, which misses the most common real problem: the GPU is fine and the host cannot keep it fed.

If your data pipeline, dataloader or storage cannot supply batches fast enough, your expensive accelerator waits. The symptom is distinctive and easy to read: GPU utilization that swings between high and near zero rather than sitting steady. That is not a GPU fault and no amount of GPU testing will find it.

To test both together, run stress-ng against the CPU and system memory while a GPU load runs, and watch whether GPU utilization holds. Better still, run your actual dataloader against your actual storage and watch the same number. If utilization is steady and high, the host is keeping up. If it sawtooths, the bottleneck is upstream of the card and a faster GPU will not help.

We cover the diagnosis in more depth in our guide to choosing a GPU, which includes how to tell memory-bound from compute-bound work.

Reading the result

Healthy. Clocks climb and hold near boost. Temperature rises and plateaus. Power draw sits near the card’s limit. No errors. Utilization steady at or near 100%.

Thermally or power limited. Clocks drop noticeably and stay down while temperature or power sits at the cap. Often normal behavior at the top end of a card’s range rather than a fault, but it tells you the sustained performance you will actually get, which is lower than the specification implies.

Faulty. Uncorrectable ECC errors, a climbing retired-page count, gpu-burn reporting incorrect results, or the process dying. On rented hardware the fix is not to debug it. Stop the pod and deploy another one. That is the advantage of renting, and it takes a minute.

Fine but not what you needed. Everything passes and the card is simply slower than your job requires. That is a sizing answer, not a health answer, and the GPU comparison guide is the better tool for it.

When you do not need to bother

Short interactive work. If you are iterating in a notebook for twenty minutes, you will notice a problem faster than a test would find it.

Serverless inference. You are not managing a specific machine, so there is nothing to test in the sense meant here.

Every single pod, every time. Checking before a long or expensive run is sensible. Checking before every five-minute job is superstition that costs more than it saves.

Doing this on Runpod

Per-second billing means a ten-minute check costs a fraction of a cent on most cards and single-digit cents on the largest. An RTX 4090 is {{gpu:rtx-4090}}/hr on Secure Cloud, or {{gpu:rtx-4090:community}}/hr on Community Cloud, and an H100 SXM is {{gpu:h100-sxm}}/hr. Current rates are on the pricing page.

If a machine looks wrong, stop it and deploy another. There is no support ticket and no waiting, which is the practical difference between renting and owning when hardware misbehaves.

For multi-node work, the test that matters most is the one between nodes rather than inside them. Clusters run 1600 to 3200 Gbps between nodes, and our guide to NVLink, InfiniBand and Ethernet explains why that number decides whether your run scales.

Clusters also save you most of the work above. They stream DCGM telemetry to a built-in Grafana dashboard at five-second granularity with no setup, which covers the signals this article tells you to watch by hand: clocks and clock event reasons, temperature against the card’s own slowdown threshold, power and thermal violations, ECC errors, row remapping and the PCIe replay counter. On a cluster you read those from a dashboard rather than a second terminal.

FAQ

How long should a GPU stress test run?

Five to fifteen minutes for a health check. Anything shorter tests the card cold, before thermal behavior shows up. Longer runs of an hour or more are for chasing intermittent faults, and are rarely worth it on rented hardware where deploying a different machine is faster than diagnosing this one.

What temperature is too hot for a GPU?

There is no single number; limits differ by card and the card enforces its own. The more useful signal is behavior: if clocks drop sharply and stay down while temperature sits at its cap, you are thermally limited. Watch the clocks rather than the thermometer.

How do I stress test a GPU and CPU at the same time?

Run a GPU load and a CPU load together, for example gpu-burn alongside stress-ng, and watch GPU utilization. Steady high utilization means the host is keeping the GPU fed. Utilization that swings between high and near zero means the bottleneck is the CPU, the dataloader or storage, not the GPU.

Can a stress test damage a GPU?

Not on server hardware in normal use. Cards enforce their own temperature and power limits and will throttle or shut down before damage. A stress test pushes a card to the state it is designed to sustain. On rented hardware you are also not the one carrying the risk.

What does it mean if my GPU throttles under load?

That it hit a temperature or power limit and reduced clocks to stay inside it. Some throttling under sustained load is normal, particularly on dense multi-GPU nodes. Persistent heavy throttling means the sustained performance you get will be below the card’s headline specification, which is worth knowing before you plan around the specification.

Is nvidia-smi a stress test?

No. nvidia-smi reports what a GPU is doing; it does not generate load. Use it alongside a tool that does. nvidia-smi dmon giving a rolling view of clocks, temperature and power is how you read the result of the test.

Get started

The quickest way to know a machine is healthy is to spend ten minutes proving it. Runpod bills per second with no minimum, so the check costs almost nothing and a bad machine can be swapped for another in about a minute. See current pricing or deploy a Pod.