How to Run LMDeploy on Runpod

LMDeploy's TurboMind engine claims 30% higher request throughput than vLLM in InternLM's own benchmarks, but most tutorials stop at a bare docker run command and never get to the flags that make that throughput real: KV cache ratio, tensor parallelism, and the quantization format that actually matches your checkpoint.

This guide covers deploying LMDeploy on Runpod, from GPU selection through the flags that determine throughput, using InternLM2.5-7B-Chat as the reference model on an L40S.

1. Why LMDeploy, and which backend to pick

LMDeploy ships two inference backends. TurboMind is a C++/CUDA engine built for maximum throughput on NVIDIA GPUs, and takes precedence by default. PyTorch Engine is pure Python, supports a broader range of model architectures, and is easier to debug when something goes wrong. Unless you're running a model TurboMind doesn't support, start with TurboMind.

GPU-to-model mapping for Runpod SKUs:

• RTX 4090 (24 GB): 7B models at BF16, or larger models with AWQ 4-bit
• L40S (48 GB): 7B at BF16 with a large KV cache budget, this tutorial's reference GPU
• A100 80GB or H100 SXM: 13B-34B at BF16, or DeepSeek-V3-class MoE checkpoints with tensor parallelism across multiple cards

2. Launching a Runpod Pod with LMDeploy

From Pods > GPU Cloud, deploy a new Pod on an L40S. Under Container Image, enter:

openmmlab/lmdeploy:latest

Attach a Network Volume of at least 30 GB, mounted at /root/.cache/huggingface, so model weights persist across pod restarts instead of re-downloading.

Under Expose Ports, add port 23333, LMDeploy's default API server port. Runpod's proxy will serve it at https://{POD_ID}-23333.proxy.runpod.net.

Set HUGGING_FACE_HUB_TOKEN as an environment variable if you're pulling a gated model, and set the pod's start command to:

lmdeploy serve api_server internlm/internlm2_5-7b-chat --backend turbomind --server-port 23333

LMDeploy's official Docker documentation runs this with --ipc=host in a plain Docker context. Runpod doesn't expose a separate IPC or shared-memory setting in the console; if you build a custom image and hit CUDA IPC errors under concurrent load, check df -h /dev/shm inside the running pod first.

3. Configuring LMDeploy for production

KV cache ratio. --cache-max-entry-count controls the proportion of free GPU memory reserved for the KV cache after model weights load. Default is 0.8. If you see an OOM error on startup, lower it:

lmdeploy serve api_server internlm/internlm2_5-7b-chat --backend turbomind --cache-max-entry-count 0.6 --server-port 23333

Tensor parallelism. For a model that doesn't fit on one GPU, --tp must equal the number of GPUs on the pod. For DeepSeek-V3-class MoE models at AWQ INT4, LMDeploy's published guidance puts five H100 SXM (400 GB combined) or four H200 SXM (564 GB combined) as the practical floor:

lmdeploy serve api_server deepseek-ai/DeepSeek-V3-AWQ --backend turbomind --model-format awq --tp 5

Quantization. For AWQ checkpoints, add --model-format awq. TurboMind also supports MXFP4 on GPT-OSS-class MoE models via --quant-policy 4, which is where TurboMind's throughput advantage over vLLM is most pronounced.

4. Testing the endpoint

LMDeploy's api_server exposes an OpenAI-compatible /v1/chat/completions route:

curl https://{POD_ID}-23333.proxy.runpod.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "internlm/internlm2_5-7b-chat",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 100
  }'

Or with the OpenAI Python SDK, overriding base_url and passing any string as the API key.

5. Monitoring and scaling

LMDeploy exposes Prometheus metrics on /metrics. The two worth alerting on: lmdeploy_kv_cache_usage above 90% signals OOM risk on long prompts, and lmdeploy_requests_waiting as a rising queue depth signals it's time to scale out. Because api_server is stateless at the HTTP layer, scaling out means launching additional LMDeploy pods, each with its own GPU allocation, behind a load balancer, or moving to Runpod Serverless if your traffic is bursty rather than steady.

6. Quick-start checklist

1. Create an L40S Pod with openmmlab/lmdeploy:latest, a Network Volume at /root/.cache/huggingface, and port 23333 exposed
2. Set the start command to lmdeploy serve api_server internlm/internlm2_5-7b-chat --backend turbomind --server-port 23333
3. Lower --cache-max-entry-count if you hit OOM on startup
4. Add --tp matching your GPU count for models too large for one card
5. Test with the curl command above

Deploy on Runpod →