Batch Inference is generally available on Runpod

Batch Inference is now available on Runpod Serverless. You can submit jobs in bulk through the same queue-based endpoint you use for real-time traffic. Batch jobs run your model and container on dedicated batch workers, so they don’t take capacity from live requests.

Start times aren’t guaranteed, so batch is for work that can wait. You can send up to 3 million requests a day, and limits depend on your account. Batch requests are billed at up to 30% off on-demand pricing, depending on GPU type. On H100, H200, B200 and B300, batch requests are billed at 20% off.

‍Bulk jobs hit the same worker cap as live traffic

Let’s say you’re running a 40,000-document embedding backfill. It’ll hit your endpoint’s max workers limit even though no one’s reading the results until morning. The cap is there to keep interactive requests responsive, and an overnight pipeline doesn’t need that.

On Runpod, real-time and batch share one endpoint, and batch runs on dedicated workers.

How teams run batch today

Most teams run batch in a few ways. The first four mean running a second setup next to your real-time stack, with its own retry logic and its own place to debug when something breaks. Batch on a Runpod endpoint is the exception.

‍Batch runs on the endpoint you already have

You don’t need to set up a second system. A batch goes to a queue-based endpoint you already run, so it uses the same model and container as your real-time traffic. Endpoints run on Runpod’s Serverless execution model, which scales workers with demand.

Requests are JSON, pasted into the console or sent in the API call, so there is no input file to upload and no output bucket to read. The console dialog is a JSON editor. Batch runs on dedicated workers that are separate from your /run traffic, so a large job doesn’t take workers away from interactive requests.

Four calls take a batch from submission to results

Nothing runs until you finalize. In the console, the Run this batch button does that step, and your endpoint page lists every batch. The docs have the full API walkthrough.

Before you start, export your endpoint ID and API key, and install jq so the first call can save the batch ID for the later ones.

export ENDPOINT_ID="your_endpoint_id"
export RUNPOD_API_KEY="your_api_key"
API="https://api.runpod.ai/v2/$ENDPOINT_ID"

# 1. Create a batch with your requests and capture the batch ID.
BATCH_ID=$(curl -s "$API/batch" \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"input": {"text": "The quick brown fox"}}, {"input": {"text": "Jumped over the lazy dog"}}]' \
| jq -r '.id')

# 2. Finalize the batch. A draft batch never runs.
curl -X POST "$API/batch/$BATCH_ID/finalize" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

# 3. Check progress.
curl -s "$API/batch/$BATCH_ID" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

# 4. Read results, 50 per page. Use offset and limit to page through, and hasMore to see if more remain.
curl -s "$API/batch/$BATCH_ID/requests?offset=0&limit=50" \
  -H "Authorization: Bearer $RUNPOD_API_KEY"

The poll call returns a summary like this (example response):

{
  "id": "batch_01j9abc123",
  "status": "FINALIZED",
  "requestTotal": 1000,
  "requestInProgress": 8,
  "requestCompleted": 244,
  "requestFailed": 6
}

A batch stays FINALIZED while it runs, so read the counts. All of its requests have finished when requestCompleted plus requestFailed equals requestTotal. This one-liner prints true when they have:

curl -s "$API/batch/$BATCH_ID" -H "Authorization: Bearer $RUNPOD_API_KEY" | jq '.requestCompleted + .requestFailed == .requestTotal'

Each result carries its own status, and a failed request carries its own error message. A batch that fails stops with the status FAILED, so check the request-level errors first. The filter lists the failed requests on a page:

curl -s "$API/batch/$BATCH_ID/requests?offset=0&limit=50" \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  | jq -c '.requests[] | select(.status=="FAILED") | {id, error}'

A batch can hold thousands of results, so page through them and keep going while hasMore is true:

OFFSET=0
while :; do
PAGE=$(curl -s "$API/batch/$BATCH_ID/requests?offset=$OFFSET&limit=50" \
 -H "Authorization: Bearer $RUNPOD_API_KEY")
 echo "$PAGE" | jq -c '.requests[]' >> results.jsonl
 [ "$(echo "$PAGE" | jq -r '.hasMore')" = "true" ] || break
 OFFSET=$((OFFSET + 50))
done

A batch holds up to 5,000 requests, and an endpoint runs up to 10 active batches, so you can send up to 50,000 requests a day. To add requests before you finalize, POST them to /batch/$BATCH_ID/requests. Each call accepts up to 10 MiB. Contact support to raise your limits, up to 3 million requests a day.

Batch is billed at up to 30% off, and cancelling stops charges

Billing is based on compute time per request. Batch requests are billed at up to 30% off on-demand pricing, depending on GPU type. On H100, H200, B200 and B300, batch requests are billed at 20% off. If you cancel a batch, queued requests aren’t billed, while in-progress requests finish and are billed. To cancel, POST to /batch/$BATCH_ID/cancel. We don’t show a cost estimate before you finalize, so run a 100-request test batch first and extrapolate.

Batch trades start time for lower cost

Runpod schedules batch workers on queue urgency and off-peak capacity, so start times aren’t guaranteed and requests run in no set order. Plan for multi-hour latency, and keep /run and /runsync for any request a user is waiting on. Console Inbox notifications, and webhook events if you’ve subscribed to batch events, fire when a batch finishes, fails or is cancelled.

Use batch for these workloads, and keep /run for the rest

Embeddings over a corpus, evaluation runs, dataset scoring, overnight processing and backfills over an existing image or document library all fit. So does a launch-day backlog, when a batch absorbs the bulk volume and your real-time endpoint serves live traffic.

‍What to do next

Running a Serverless endpoint? Read our Batch Inference documentation and submit a 100-request test batch.

Need higher limits? Contact support to raise them to up to 3 million requests a day.

New to Runpod? Deploy a Serverless endpoint, then try Batch Inference.