How to Run LocalAI on Runpod

LocalAI's pitch is a single OpenAI-compatible server for text, image, audio, and vision models with no GPU strictly required, but the "no GPU required" framing is exactly why people under-provision it on a real deployment and then wonder why a 13B model runs at CPU speed on a rented GPU pod. This guide covers deploying LocalAI on Runpod correctly the first time.

1. Picking a GPU and the right container image

LocalAI publishes separate images per accelerator, and using the wrong one means silently falling back to CPU inference:

localai/localai:latest-gpu-nvidia-cuda-12 for CUDA 12 (broadest compatibility as of mid-2026)
localai/localai:latest-gpu-nvidia-cuda-13 for CUDA 13
localai/localai:latest-aio-gpu-nvidia-cuda-12, the "all-in-one" variant that bundles pre-selected backends and models for immediate use rather than an empty gallery

For a general-purpose chat and embeddings server, an L40S (48 GB) comfortably runs 7B-13B models with headroom for LocalAI's other backends (Whisper for transcription, Stable Diffusion for image generation) running alongside. For lighter single-model deployments, an RTX 4090 (24 GB) is sufficient.

2. Launching a Runpod Pod with LocalAI

From Pods > GPU Cloud, deploy on an L40S with:

localai/localai:latest-gpu-nvidia-cuda-12

Attach a Network Volume of at least 30 GB, mounted at /models, so downloaded models survive pod restarts. Set the environment variable:

MODELS_PATH=/models

Under Expose Ports, add port 8080. Runpod's proxy serves it at https://{POD_ID}-8080.proxy.runpod.net.

Leave the start command at the image default, or explicitly install a model on boot by passing a model name as an argument (for example, a small instruct model to verify the deployment works before installing anything larger).

3. Installing a model

LocalAI's gallery system installs models without a separate download-and-configure step. Once the pod is running, open https://{POD_ID}-8080.proxy.runpod.net in a browser, go to the Models page, search for the model you want, and click Install. Alternatively, install via API:

curl https://{POD_ID}-8080.proxy.runpod.net/models/apply \
  -H "Content-Type: application/json" \
  -d '{"id": "[email protected]"}'

Because models download to the Network Volume, a second pod pointed at the same volume boots with the model already present.

4. Verifying GPU acceleration is actually active

This is the step most LocalAI deployments skip. Confirm the container has GPU access before assuming inference is accelerated:

docker run --rm --gpus all nvidia/cuda:12.8.0-base-ubuntu24.04 nvidia-smi

Then, while running a real inference request against your LocalAI endpoint, watch GPU utilization from inside the pod:

watch -n 1 nvidia-smi

If GPU-Util stays at 0% during a request, the model loaded on the CPU backend rather than GPU. Check the model's YAML configuration for the correct backend field, or confirm you pulled a GPU-tagged image rather than the CPU-only tag.

5. Testing the API endpoint

LocalAI implements OpenAI-compatible endpoints for chat, embeddings, and audio:

curl https://{POD_ID}-8080.proxy.runpod.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Check readiness first with curl https://{POD_ID}-8080.proxy.runpod.net/readyz, which returns 200 once a model is loaded and the server can accept requests.

6. Securing an exposed LocalAI instance

LocalAI does not require an API key by default. If you're exposing the Runpod proxy URL beyond your own testing, set an API key via the API_KEY environment variable before treating the endpoint as production-ready; without it, anyone with the URL can run inference billed to your pod.

7. Quick-start checklist

Create an L40S Pod with localai/localai:latest-gpu-nvidia-cuda-12, a Network Volume at /models, and port 8080 exposed
Set MODELS_PATH=/models as an environment variable
Install a model via the web UI or the /models/apply endpoint
Confirm GPU utilization rises during inference with nvidia-smi
Set API_KEY before exposing the endpoint beyond local testing

Deploy on Runpod →