How to Run TGI (Text Generation Inference) on Runpod

Most TGI tutorials stop at docker run and a quick generation test. They don't cover GPU sizing, the shared-memory flag that silently breaks multi-GPU setups, the token budget that determines whether your server handles 10 concurrent requests or 200, or how to keep model weights from re-downloading every time you restart the container.

This guide covers the complete workflow for deploying Hugging Face's Text Generation Inference (TGI) on Runpod, from GPU selection through production tuning, using Llama 3.1 8B Instruct as the reference model and an L40S as the default GPU. The result is a running, OpenAI-compatible inference server you can point real traffic at.

1. Prerequisites and GPU selection for TGI on Runpod

Before you start, you need:

• A Runpod account with billing configured
• A Hugging Face token with access to meta-llama/Meta-Llama-3.1-8B-Instruct (request access at hf.co/meta-llama)
• Basic familiarity with Docker CLI

VRAM is the first constraint to size correctly. Weight-only memory for common model sizes:

These are floor numbers. TGI's token budget (covered in Section 4) adds KV cache on top, which is why --max-total-tokens matters as much as the weights themselves.

GPU-to-model mapping for Runpod SKUs:

• RTX 4090 (24 GB): 7B fp16 or 13B with 4-bit quantization
• L40S (48 GB): 7B fp16 comfortably with room for a large token budget, or 13B fp16. The right default for most 7-13B deployments and this tutorial's reference GPU
• A100 80GB or H100 PCIe (80 GB): 13B fp16 at high concurrency, or 34B quantized
• H100 SXM (80 GB, multi-GPU with NVLink): 70B fp16 with --num-shard 2 across two cards

Runpod provisions all of these GPU types on-demand with per-second billing on Pods, which matters when you're iterating on flags and don't want to commit to a long-running instance.

2. Launching a Runpod Pod with TGI

From the Runpod console, go to Pods > GPU Cloud and deploy a new Pod. Select L40S as the GPU type, and under Container Image enter:

ghcr.io/huggingface/text-generation-inference:3.3.5

This is the official TGI image maintained by Hugging Face. It bundles CUDA, the Rust-based TGI server, and the OpenAI-compatible Messages API. No custom build required for standard deployments.

Attach a Network Volume before launching, sized at least 50 GB (Llama 3.1 8B weights are roughly 16 GB; leave headroom for the tokenizer and any additional models you cache later). Mount it at:

/data

TGI's official quickstart uses this path as the shared volume for model weights. On first boot, TGI downloads weights here; on every subsequent restart with the same volume attached, it loads from disk instead of re-downloading, cutting cold start from minutes to seconds.

Under Expose Ports, add port 80 (TGI's internal listen port). Runpod's reverse proxy will generate a public HTTPS endpoint at https://{POD_ID}-80.proxy.runpod.net.

Add your Hugging Face token as an environment variable named HF_TOKEN. TGI relies on CUDA IPC for inter-process communication, which needs more headroom than Docker's bare 64 MB /dev/shm default under concurrent load. Runpod doesn't expose a separate shared-memory field in the console. If you're using a custom base image and see CUDA IPC errors appear only under concurrent load (not on the first request), check df -h /dev/shm inside the running pod before assuming it's a code issue.

Set the pod's start command arguments to:

--model-id meta-llama/Meta-Llama-3.1-8B-Instruct

TGI v3 ships with zero-config mode: leave the token budget flags unset and it auto-detects the maximum safe values for your hardware at startup. That's the right default until you have a specific reason to override it (Section 4 covers when that is).

3. Writing a Dockerfile for a custom TGI image

The official image covers most use cases. You need a custom Dockerfile when:

• You're loading weights from a private registry rather than the Hugging Face Hub
• You have additional Python dependencies, such as custom tokenizers or internal packages
• You want to pin to a specific TGI release for reproducibility across environments

FROM ghcr.io/huggingface/text-generation-inference:3.3.5

# Add any custom dependencies
COPY requirements-extra.txt .
RUN pip install --no-cache-dir -r requirements-extra.txt

# Optional: bake in a private model or config
# COPY ./custom-config /custom-config

Build and push to Docker Hub or GitHub Container Registry, then reference your image instead of the official tag in the Runpod pod template.

4. Configuring TGI flags for production

Zero-config mode is a reasonable default, but production traffic usually needs explicit control over two things: the token budget and quantization.

Token budget. TGI plans batching around two caps:

• --max-input-tokens: the longest prompt TGI will accept
• --max-total-tokens: prompt plus generated tokens combined, which bounds per-request memory

For an L40S running Llama 3.1 8B with typical chat-length prompts, --max-input-tokens 4096 --max-total-tokens 8192 is a reasonable starting point. Push these higher only if your workload genuinely needs longer context; every increment shrinks the KV cache budget available for concurrent requests.

Quantization. For AWQ or GPTQ pre-quantized checkpoints:

--model-id TheBloke/Llama-2-13B-chat-AWQ --quantize awq

For on-the-fly 4-bit quantization of an unquantized checkpoint using bitsandbytes:

--model-id meta-llama/Meta-Llama-3.1-8B-Instruct --quantize bitsandbytes-nf4

The distinction matters: --quantize awq or --quantize gptq require a checkpoint that's already quantized in that format. TGI won't convert an fp16 checkpoint to AWQ on load. bitsandbytes is the only option that quantizes at load time from a standard checkpoint.

Multi-GPU deployment with tensor parallelism

For a 2x A100 80GB pod running a 70B model:

--model-id meta-llama/Meta-Llama-3.1-70B-Instruct --num-shard 2

--num-shard must equal the number of GPUs attached to the pod. TGI shards model weights across them and uses NCCL for cross-GPU communication, which is more shared-memory-sensitive than single-GPU inference, another reason to check /dev/shm first if a multi-GPU pod fails only under load.

5. Testing the TGI API endpoint

Once the pod is running, find the proxy URL under the pod's Connect tab. It follows the pattern https://{POD_ID}-80.proxy.runpod.net.

Confirm the server has finished loading by checking the Logs tab for a message indicating the server is ready to accept connections, or poll the health route directly.

Test with the OpenAI-compatible Messages API:

curl https://{POD_ID}-80.proxy.runpod.net/v1/chat/completions \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tgi",
    "messages": [
      {"role": "user", "content": "What is deep learning?"}
    ],
    "max_tokens": 128
  }'

The OpenAI Python SDK works unchanged against this endpoint; override base_url and pass any string as api_key (TGI does not enforce API key auth by default, so put the proxy URL behind your own auth layer before exposing it publicly).

Three common failure modes

1. CUDA OOM on startup. The model won't load if VRAM is insufficient. Lower --max-total-tokens, switch to a quantized checkpoint, or move up a GPU tier.

2. Port not reachable. Confirm port 80 is declared under the pod's exposed ports, not just referenced in your Docker command. Runpod's proxy only routes to declared ports.

3. Gated model download fails with a 401. Verify HF_TOKEN is set as an environment variable on the pod and that your Hugging Face account has accepted the model's license at hf.co.

6. Tuning TGI for production traffic

TGI's continuous batching groups in-flight requests dynamically rather than processing them strictly one at a time, which is the core of its throughput advantage over a naive request queue. It also streams tokens via server-sent events for responsive chat UIs.

TGI exposes Prometheus metrics on /metrics, covering queue depth, request latency, and batch-level timing. The two numbers worth watching in production:

• Queue size: sustained growth means the server is saturated for its current concurrency ceiling
• Batch token utilization: consistently near the --max-total-tokens ceiling signals it's time to raise the budget or add a worker

Persistent Pod vs. Runpod Serverless: billing fit

A Pod bills per second regardless of whether it's actively serving requests, which suits steady, predictable traffic well. If your load is bursty, with real gaps between peaks, Runpod Serverless is the better architectural fit: it scales workers to zero between requests and bills only for active processing time. Moving from a Pod to Serverless means wrapping the same TGI image in a handler function rather than a direct drop-in, but the container and flags carry over unchanged.

7. Deploy TGI on Runpod: quick-start checklist

1. Create an L40S Pod on Runpod with ghcr.io/huggingface/text-generation-inference:3.3.5, attach a 50 GB Network Volume at /data, expose port 80
2. Set HF_TOKEN as an environment variable
3. Set the start command to --model-id meta-llama/Meta-Llama-3.1-8B-Instruct
4. Wait for the model to download on first boot, or restart with the same Network Volume for a warm cache
5. Hit the proxy URL with the curl command from Section 5

That's the complete TGI-on-Runpod workflow. If your traffic is bursty, wrap the same image for Runpod Serverless. If you need multi-GPU scale, add --num-shard matching your GPU count and select a multi-GPU pod.

Deploy on Runpod →