How to Run LiteLLM on Runpod

LiteLLM doesn't run inference itself. It's a proxy that gives you one OpenAI-compatible endpoint in front of OpenAI, Anthropic, Bedrock, and any self-hosted model you're already running, which makes the useful question not "how much GPU does LiteLLM need" (none) but "how do I put it in front of a model I'm already hosting on Runpod." This guide covers both: deploying the LiteLLM proxy itself, and wiring it to a Runpod-hosted vLLM or TGI endpoint alongside your other providers.

1. LiteLLM doesn't need a GPU

The proxy is a Python/Rust gateway that routes requests and tracks spend; the actual token generation happens wherever the target model lives. Run it on a small Runpod CPU pod, or colocate it on the same pod as a lightweight GPU workload if you don't want to manage two deployments. Don't put it on an expensive GPU tier; it won't use the GPU.

2. Launching the LiteLLM proxy on Runpod

From Pods > CPU (or a GPU pod if you're colocating), deploy with:

docker.litellm.ai/berriai/litellm:main-v1.90.2

Pin a specific version tag. LiteLLM had a supply chain incident in March 2026 affecting versions 1.82.7 and 1.82.8; the fix shipped in 1.83.0. Never deploy :latest in production, here or anywhere: pin a tag you've verified and control your own upgrade timing.

Write a minimal config.yaml mapping a friendly model name to your Runpod-hosted vLLM endpoint alongside a cloud provider:

model_list:
  - model_name: my-runpod-llama
    litellm_params:
      model: openai/meta-llama/Meta-Llama-3.1-8B-Instruct
      api_base: https://{POD_ID}-8000.proxy.runpod.net/v1
      api_key: "placeholder"
  - model_name: claude-fallback
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

The openai/ prefix on the Runpod model tells LiteLLM to treat it as an OpenAI-compatible endpoint rather than route it through a specific provider SDK, since your self-hosted vLLM or TGI server already speaks that API.

Mount the config and set your master key as environment variables. Under Expose Ports, add port 4000:

docker run -v $(pwd)/config.yaml:/app/config.yaml \
  -e ANTHROPIC_API_KEY=your-key \
  -e LITELLM_MASTER_KEY=sk-your-master-key \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:main-v1.90.2 \
  --config /app/config.yaml

Runpod's proxy serves the admin UI and API at https://{POD_ID}-4000.proxy.runpod.net.

3. Why route a self-hosted model through LiteLLM instead of hitting it directly

Once your Runpod-hosted vLLM or TGI endpoint sits behind LiteLLM alongside your cloud providers, you get, for free: spend tracking per model and per team, automatic fallback if the self-hosted endpoint is cold-starting or down (fallbacks: [{"my-runpod-llama": ["claude-fallback"]}] in the config), and virtual keys so different applications or teams get scoped access without sharing your Runpod endpoint's raw URL. Your application code targets one base URL and swaps models by name, whether the request lands on your own GPU or a cloud provider.

4. Persistent config with Postgres

The minimal setup above stores config in a flat file. For production, add Postgres so you can manage models and generate virtual keys from the admin UI at runtime instead of editing YAML and restarting:

docker run -v $(pwd)/config.yaml:/app/config.yaml \
  -e DATABASE_URL="postgresql://user:pass@host:5432/litellm" \
  -e LITELLM_MASTER_KEY=sk-your-master-key \
  -e LITELLM_SALT_KEY=your-random-salt-value \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm-database:main-v1.90.2 \
  --config /app/config.yaml

LITELLM_SALT_KEY encrypts provider credentials stored in the database. Generate it once and never change it; rotating it makes previously stored credentials unreadable.

5. Testing the endpoint

curl https://{POD_ID}-4000.proxy.runpod.net/chat/completions \
  -H "Authorization: Bearer sk-your-master-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-runpod-llama",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Or with the OpenAI Python SDK: point base_url at your proxy URL and use a LiteLLM virtual key as api_key. The admin UI is available at https://{POD_ID}-4000.proxy.runpod.net/ui, username admin, password your master key.

6. Quick-start checklist

Deploy a Runpod Pod with a pinned LiteLLM image tag (never :latest)
Write a config.yaml pointing at your Runpod-hosted model with the openai/ prefix, plus any cloud providers
Set LITELLM_MASTER_KEY, expose port 4000
Add Postgres and LITELLM_SALT_KEY if you want runtime model management via the admin UI
Test with the curl command above

Deploy on Runpod →