Runpod Serverless gives you load-balanced endpoints, automatic worker scaling, streaming and WebSockets. It does not provide traffic splitting, gRPC, shared GPU pools across services, or per-worker memory limits. This guide covers both halves, because knowing what a platform will not do is usually the faster way to decide whether it fits.
What Runpod Serverless gives you
Load balancing endpoints distribute requests across workers automatically. Each worker runs your container, and the platform scales worker count between the minimum and maximum you set.
Concurrency. max_concurrency controls how many requests one worker handles at a time. It defaults to 1, which is the right setting for most GPU inference, where a single request already saturates the card.
Scaling. There are two scaler types. QUEUE_DELAY adds workers when requests wait longer than your threshold. REQUEST_COUNT adds workers based on requests in flight. Combine either with min and max workers to set the floor and the ceiling.
Timeouts. execution_timeout_ms caps how long a single request may run before the platform reclaims the worker.
Streaming. Supported, and the usual choice for chat-style responses where first-token latency matters more than total time.
WebSockets. Supported, for persistent bidirectional connections such as voice agents or interactive sessions.
Multi-GPU workers. A worker can use one GPU or several, and a single model can be spread across multiple GPUs in parallel. This is how you serve a model too large for one card.
Rollback. GitHub release rollback moves an endpoint back to a previous release. It is a deployment control rather than a routing control, and that distinction matters below.
What Runpod does not do
Traffic splitting between model versions
There is no weighted routing, canary release or percentage split across versions on a single endpoint.
The closest supported mechanism is GitHub release rollback, which switches an endpoint wholesale from one release to another. That covers shipping a version and reverting it if it breaks. It does not cover sending five percent of traffic to a new version.
If you need a split, run the versions as separate endpoints and divide traffic in your own application or gateway. You control the ratio and can change it without redeploying either endpoint.
gRPC
Serverless endpoints are TCP only. There is no HTTP-over-gRPC transport, and UDP is not available either.
Use HTTP instead. For token-by-token responses, use the streaming endpoint rather than reaching for gRPC bidirectional streaming. For persistent connections, WebSockets are supported.
Several services sharing one GPU pool
One endpoint serves one model. Separate services cannot be packed onto a shared set of cards and scheduled against each other.
A note on the term, because it causes confusion. Runpod uses "GPU pool" internally to mean a group of cards within one SKU, for example a 24GB SKU holding three cards. That is a capacity grouping, not a multi-tenant scheduler.
What you can do instead. One model can span several GPUs inside a worker, so pooling for a single large model is supported. For several models, run several endpoints. Because workers scale to zero, idle endpoints cost nothing, which removes most of the reason to pack models together in the first place.
Per-worker memory limits
There is no setting that caps host or GPU memory per worker.
You size at the GPU tier instead, and for GPU inference that is usually the better control anyway. Choose the SKU whose VRAM fits your model and its working set, then use max_concurrency to limit how many requests share that memory at once. A memory cap would mostly duplicate what those two already decide: the card sets the ceiling, and concurrency sets how much of it any one request can claim.
Where a per-worker limit earns its keep elsewhere is packing several workloads onto one machine, and that is not how Runpod Serverless allocates GPUs.
Choosing a serving pattern
FAQ
Does Runpod support gRPC?
No. Serverless endpoints are TCP only, with no HTTP-over-gRPC transport and no UDP. Use HTTP, with the streaming endpoint for token-by-token responses and WebSockets for persistent connections.
Can I split traffic between two model versions?
Not on a single endpoint. Runpod has no weighted routing or canary release. Run the versions as separate endpoints and split traffic in your own gateway, or use GitHub release rollback if you only need to ship and revert.
Can several models share one GPU pool?
No. One endpoint serves one model. A single model can be spread across several GPUs inside a worker, but separate services cannot share a set of cards. Since endpoints scale to zero, running one per model does not cost you anything while idle.
Can I set a memory limit per worker?
No. Choose the GPU SKU whose VRAM fits your model and use max_concurrency to control how many requests share it. Together those do what a memory limit would.
