How Runpod Serverless billing works when a job fails to complete

Let's say you've been hard at work implementing a chatbot using Runpod serverless, and after an hour of testing you can see your balance chipping away a bit, even though you know you aren't using any storage and don't have any stopped pods, even though you haven't actually had a successful request served yet. That's a common surprise, and an understandable one. From the outside, a failed request looks like nothing happened. However, on the platform side, a GPU was allocated, a container started, a model loaded into memory, and your code ran. All of that is compute time, and it's worth a breakdown of where your spend is going.

If you take one idea from this post, make it this one:

Serverless billing follows worker time, not request outcomes. You pay for the seconds a worker is running on a GPU (starting up, executing, or waiting for more work), whether the jobs it touched succeeded, failed, or never started.

Below, we walk through the billable lifecycle of a worker, the places where "failed" and "billed" overlap, how to trace a surprise charge back to its cause, and the safeguards that stop a bad deploy from turning into a bad bill. We finish with what to do if you think you've been charged incorrectly.

Key takeaways

• A worker is billed per second from when it starts until it stops. That covers startup (the cold start), execution, and the idle window after each job. Active workers are billed for as long as they're enabled.
• A job's status (COMPLETED, FAILED, TIMED_OUT) tells you what happened to the work, not what you were charged. A failed job that held onto that GPU for ten minutes costs the same as a job that returned a successful result.
• The expensive failures are the ones that repeat: workers that crash during startup while jobs wait in the queue, handlers that hang until the execution timeout, and requests that get retried by the platform or by your client.
• Your maximum burn rate is Max workers * GPUs per worker * your GPU's rate, plus any active workers. You'll want to be diligent about what you set in thee variables. During your testing and iteration phases, you'll want to use as few workers and as cheap a GPU spec as you can technically get away with.
• To stop spend right away, set Max workers and Active workers to 0, then purge the queue.
• If you reconcile your usage and the numbers still don't add up, contact support with your endpoint IDs, a UTC time window, and job and worker IDs.

1. The billable lifecycle of a Serverless worker

A Serverless endpoint is a queue (or, for load-balanced endpoints, a router) that sits in front of a pool of workers. Each worker is a container running your image on a GPU. The platform scales the pool up and down for you, between the Active workers floor and the Max workers ceiling you set. Flex workers start when there's work and shut down when the idle timeout expires. You pay only while they're running. Active workers run around the clock so requests never wait on a cold start.

Some details that matter in practice:

• Billing is per second. Short bursts cost little, which is the joy of Serverless. However, a small cost repeated thousands of times does not.
• Cold starts are real compute. If your container downloads a 20 GB model every time it starts, you pay for that download every time. FlashBoot (on by default, at no extra cost), cached models, network volumes, and baking weights into the image all make this shorter.
• The idle window restarts after every job. At the default 5-second idle timeout, a steady trickle of requests can keep a worker alive indefinitely. That's normally what you want. It stops being what you want when every one of those requests is failing.

Load-balanced endpoints follow the same rule: you pay for the time workers are running. The difference is that there's no job queue. Requests go straight to healthy workers, so the health check your worker exposes decides whether it gets traffic. A worker that reports healthy while your app is broken will keep receiving requests and keep running.

2. Execution, success, retries, startup failures, and what your users see

Most billing confusion comes from the different signals and how they relate to what you actually pay.

hese combinations catch people out regularly:

• FAILED, and billed in full. Say your handler spends 9 minutes generating a video and then throws on the final encode. The job is FAILED, and you pay for 9 minutes of execution.
• COMPLETED, but useless. If your handler catches an exception and returns {"status": "error"} as ordinary output, the platform records a successful job. Return errors under the error key instead. They'll be counted as failures, and your dashboards will show what actually happened.
• Timed out at the client, still running on the GPU. A synchronous /runsync call returns after a set wait even when the job isn't done, and your HTTP client may give up even sooner. Neither one stops the job. It keeps running, and billing, until it finishes, hits the server-side timeout, or you cancel it.
• Retried, so billed again. If a worker dies mid-job (say, the process is killed for running out of memory), the platform can put the job back in the queue for another worker. That helps reliability, but if a request always crashes the worker, you pay for every attempt. Client-side retries add to the total: resubmitting a timed-out /run call creates a second job while the first may still be running.
• Never ready, still billed. A worker that spends 90 seconds loading and then crashes has used 90 seconds of GPU time. It never processed a job, so it won't appear in your request history. It shows up only in worker states, logs, and billing.

3. Where your balance goes when nothing seems to work

4. How to trace a surprise charge

Start at the bill and work toward the cause. First pin down where and when the money went, then work out why.

Step 1: Find the spend in your billing data

In the console, open Billing to see usage over time, broken down by resource. You want two things from it: which endpoint (or endpoints) account for the charge, and the time window where the rate changed. On a team, charges go to the team owner's account, and only members with the billing or admin role can see them.

For more detail, pull serverless billing history from the REST API. Hourly buckets for a single endpoint usually make it obvious when an incident started:

curl -s "https://rest.runpod.io/v1/billing/endpoints?endpointId=$ENDPOINT_ID&bucketSize=hour&startTime=2026-10-12T00:00:00Z&endTime=2026-10-13T00:00:00Z" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

If you prefer the command line, runpodctl billing returns the same data.

Write down the window where spend jumped, in UTC. The rest of this section is about explaining that window.

Step 2: Check endpoint health and metrics

Open the endpoint, go to the Metrics tab, and filter to your window. Compare these:

• Completed, failed, and retried requests. A high share of failures, or retries that don't stop, point to the handler and poison-request patterns in section 3.
• Execution time. Spikes, or a flat line at your execution timeout, mean jobs are hanging or running slow.
• Cold start count and cold start time. Lots of cold starts with few completed jobs is what a startup crash loop looks like. It can also mean the idle timeout is too short for bursty traffic.
• Worker counts by state. Workers running while nothing completes is the clearest sign you're paying for failure.

For a live snapshot, call the health endpoint:

curl -s "https://api.runpod.ai/v2/$ENDPOINT_ID/health" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

{
  "jobs":    { "completed": 12, "failed": 340, "inProgress": 10, "inQueue": 2210, "retried": 41 },
  "workers": { "idle": 0, "running": 10 }
}

This response shows thousands of jobs queued, ten workers running, and nearly everything failing. When you see something like it, stop the endpoint first (section 7) and diagnose afterward.

Step 3: Read the request history

The Requests tab lists recent jobs with their status, delay time, and execution time. Look for:

• TIMED_OUT jobs, and how close their execution time is to your timeout.
• IN_PROGRESS jobs that are far older than your usual execution time.
• FAILED jobs with long execution times. These are your most expensive failures.
• Repeated, near-identical inputs, which usually mean client retries.

For any single job, /status returns its state and its timings in milliseconds:

curl -s "https://api.runpod.ai/v2/$ENDPOINT_ID/status/$JOB_ID" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

{
  "id": "a1b2c3d4-...",
  "status": "FAILED",
  "delayTime": 84213,
  "executionTime": 600000,
  "error": "..."
}

Note that delay time doesn't measure billing. It's how long a job waited before a worker picked it up. Waiting in the queue with no worker costs nothing. If the delay came from a cold start, though, the worker that was starting up was billed for that time.

Step 4: Look at worker behavior

The Workers tab shows each worker's current state, and you can open its logs from there. Look for:

• Churn: lots of workers appearing and disappearing in a short time, particularly ones moving between initializing and unhealthy.
• Workers stuck in one state: initializing for much longer than your normal cold start, or running with no matching in-progress jobs.
• Outdated workers after a deploy. During a rolling update, old workers keep serving while new ones start, so for a while you pay for both. If the new release is broken, the new workers fail while the old ones drain away.

Step 5: Read the logs

Worker logs record your container's stdout and stderr from startup on. That makes them the best evidence for startup failures, which are the one category that never shows up in request history. Endpoint logs combine output from all workers.

Things to search for:

• The last lines before a worker exits: tracebacks, CUDA error, out of memory, Killed, missing files, failed downloads.
• The gap between container start and your "ready" log line. That gap is your billed cold start.
• The same job ID in more than one worker's logs, which means the job was retried.
• Error messages from downstream services your handler calls.

Logs help much more when your code writes them on purpose. Print a line when startup begins, after each slow step, and when the worker is ready for jobs, and put the job ID in every log line inside the handler. Use flush=True (or unbuffered output) so the last lines before a crash actually get written. Workers are short-lived, so if you need logs for the long term, send them to your own logging service too.

Step 6: Reconcile billed time against useful time

With the data from the steps above, you can split a window's billed time into its parts:

Here's an illustrative one-hour window for an endpoint that just shipped a broken release:

Illustrative numbers. Shares are rounded.

Only 15% of the hour produced a successful result. The fix isn't in billing. It's in the startup path (42%) and the hung jobs (25%). A breakdown like this also shows whether anything is left unexplained. If billed time is much larger than the total running time of every worker you can account for, it's time to consider a billing issue (section 8).

5. Settings that shape your spend

Every Serverless endpoint setting trades off latency, reliability, and cost. These are the ones that matter most when something goes wrong.

Per-request execution policies let you set tighter limits for specific workloads without touching the endpoint:

{
  "input": { "prompt": "..." },
  "policy": {
    "executionTimeout": 120000,
    "ttl": 600000,
    "lowPriority": true
  }
}

‍

executionTimeout and ttl are in milliseconds. lowPriority jobs don't make the endpoint scale up, so they suit work that should only use capacity that's already running.

Work out your maximum burn rate for every production endpoint: max hourly spend:  (Max workers × GPUs per worker × flex hourly rate)+ (Active workers × GPUs per worker × active hourly rate)

6. Safeguards and alerts

Before you deploy

• Test the image before it scales. Run your handler locally with a test input (python handler.py --test_input '{"input": {...}}'), or start the image on a single Pod to make sure it loads and runs on real hardware.
• Canary new releases. Deploy to a separate endpoint, or set Max workers to 1, send a few real requests, and check the logs before you scale up.
• Pin image tags. With :latest, it's hard to know what's running and easy to ship something untested.
• Bake in or cache your model. It makes cold starts faster and removes a common cause of startup failures.

In your worker code

Make failures fast, loud, and contained. Do the expensive setup once per worker, outside the handler. Check everything you can before loading the model, and return errors instead of crashing:

import os
import sys
import runpod

# 1. Fail fast on configuration problems, before any expensive work.
REQUIRED_ENV = ["MODEL_PATH"]
missing = [k for k in REQUIRED_ENV if not os.environ.get(k)]
if missing:
    print(f"FATAL: missing environment variables: {missing}", flush=True)
    sys.exit(1)

import torch

if not torch.cuda.is_available():
    print("FATAL: no CUDA device visible", flush=True)
    sys.exit(1)

print("startup: loading model", flush=True)
model = load_model(os.environ["MODEL_PATH"])  # once per worker, not per job
print("startup: ready", flush=True)


def handler(job):
    job_id = job["id"]
    inp = job.get("input") or {}

    # 2. Reject bad input in milliseconds, not minutes.
    if "prompt" not in inp:
        return {"error": "input.prompt is required"}

    try:
        # 3. Put timeouts on anything that can block (HTTP calls, locks, subprocesses).
        result = model.generate(inp["prompt"], max_tokens=inp.get("max_tokens", 512))
        return {"output": result}
    except torch.cuda.OutOfMemoryError as e:
        print(f"[{job_id}] OOM: {e}", flush=True)
        # 4. Fail this job once, and recycle the worker so the next job starts clean.
        return {"error": f"out of memory: {e}", "refresh_worker": True}
    except Exception as e:
        print(f"[{job_id}] error: {e}", flush=True)
        return {"error": str(e)}


runpod.serverless.start({"handler": handler})

How the pattern works:

• Returning a dictionary with an error key marks the job FAILED, so your metrics match what really happened.
• refresh_worker stops the worker after the current job. Use it when an error could leave the GPU or the process in a bad state.
• For deeper startup validation, the Runpod SDK supports fitness checks that run before a worker starts taking jobs. Use them to check GPU health, available memory, and required files.

In your client

• Cancel what you abandon. When a caller gives up, call /cancel/{job_id}.
• Don't resubmit blindly. Check /status on the original job before you retry. Cap the number of retries and use exponential backoff.
• Stop sending when error rates spike. A simple circuit breaker in your client keeps fast failures from scaling into a large bill.

On your account

• Watch your balance. Turn on low-balance notifications in billing settings. If you use automatic top-ups, choose a threshold and amount that match what you'd accept spending in a bad hour. Auto-pay also removes the hard stop you'd otherwise get when the balance hits zero.
• Scope your API keys. Use separate keys for production, staging, and one-off scripts, each with the narrowest permissions that work. Then a leaked key is easy to contain.
• Limit who can change endpoint settings. On teams, give billing visibility to the people who watch costs, and edit access only to the people who deploy.

Automated monitoring and an emergency brake

The console is good for investigating, but alerts need to reach you on their own. This script polls /health and alerts you when an endpoint is clearly paying for failure. With AUTO_BRAKE=1, it also sets Max workers to 0. Run it on a schedule, from somewhere other than the endpoint it's watching.

You can add a spend check in the same way: pull the last hour from the serverless billing history API, and brake if it's over your budget.

7. Containment runbook: stopping the spend

If your balance is dropping and you can't explain why right away, stop the spend first and investigate second. A crash loop at full Max workers costs the same per minute whether or not anyone is watching.

1. Take a quick snapshot. Spend 60 seconds screenshotting the Workers and Metrics tabs and saving the /health output. Worker logs are easiest to collect while the workers still exist.
2. Set Max workers to 0 and Active workers to 0. You can do this in the endpoint's settings or through the API. This is the brake: no new workers start, and existing workers are scaled down. Jobs that are mid-run when their worker stops may not finish.
3. Purge the queue. Queued jobs will start workers again as soon as you raise Max workers. If they're not worth running, clear them:

   curl -s -X POST "https://api.runpod.ai/v2/$ENDPOINT_ID/purge-queue" \
     -H "Authorization: Bearer $RUNPOD_API_KEY"

4. Cancel the stuck in-progress jobs you can identify:

   curl -s -X POST "https://api.runpod.ai/v2/$ENDPOINT_ID/cancel/$JOB_ID" \
     -H "Authorization: Bearer $RUNPOD_API_KEY"

5. If the traffic is unexpected, shut off its source. Pause whatever upstream job or queue is sending requests, and rotate the API key if there's any chance it leaked.
6. Confirm the meter has stopped. Check that no workers are running, and that billing history shows no new usage on the endpoint over the following hour. Network volume storage keeps billing.
7. Fix it, then reopen gradually. Deploy the fix, set Max workers to 1, send test requests, and watch the logs before you restore normal limits.

Don't start by deleting the endpoint. Scaling to zero stops compute charges just as effectively, and keeping the endpoint preserves the metrics, request history, and logs that you and our support team will need.

8. Escalating a suspected billing defect

Most surprise charges turn out to be one of the patterns in section 3: real compute spent on work that didn't succeed. Billing systems can have defects too, though. If you've reconciled your usage and the numbers still don't add up, we want to hear about it.

How to escalate

1. Contain it first. Follow section 7 so the charge stops growing while you wait.
2. Check uptime.runpod.io for incidents that overlap your window. If you find one, include the link.
3. Gather the evidence listed below. The more specific you are, the faster we can match your report to platform-side records.
4. Contact Runpod support through the support option in the console, or email [email protected]. Put the endpoint ID and the time window in the first lines of your message. Billing investigations involve account details, so use support rather than public community channels.
5. Leave the endpoint and its history in place until the investigation is finished.

Support will review platform-side records for the endpoints and window you report, explain what they find, and follow up with next steps.

Evidence checklist

• Account email, and team name if billing goes to a team
• Endpoint ID(s)
• Start and end of the affected window, in UTC
• Expected spend vs. observed spend, with a billing history export for the window
• The worker IDs involved, with the states you saw and when you saw them
• A sample of job IDs, especially stuck, failed, retried, or timed-out jobs
• Relevant log excerpts, including the last lines before workers exited
• Recent changes: image tag, template, endpoint settings, traffic
• What you did to contain it, and when (for example, "Max workers set to 0 at 14:05 UTC")
• A link to any overlapping status page incident

Wrapping up

The meter on a Serverless endpoint measures one thing: how long your workers are running. Failures don't stop it. They only change what you get for that time. Every expensive pattern above has the same shape, something repeating automatically, and each one has a limit you control:

• Max workers caps the blast radius.
• The execution timeout caps hung jobs.
• The job TTL and a queue purge cap how long the autoscaler keeps trying.
• Fail-fast startup code caps what each crash costs.

A short checklist to take with you:

• Calculate your maximum burn rate and set Max workers on purpose
• Keep the execution timeout on and close to your real worst case
• Fail fast at startup, and return errors under the error key
• Cancel abandoned jobs and cap client retries
• Alert on failure rate, retries, unhealthy workers, and stalled queues
• Know the brake: Max workers to 0, purge the queue, then investigate

If you'd like to poll other users for help, feel free to check out our Discord or our Reddit; both are full of developers like you that would love to give you a hand.