How to Run Aphrodite Engine on Runpod

Aphrodite Engine is PygmalionAI's fork of vLLM built for the checkpoints mainline vLLM doesn't prioritize: EXL2, GGUF, and community GPTQ quants pulled straight off Hugging Face, plus samplers (DRY, XTC, mirostat) that matter for chat and creative-writing output quality. It keeps vLLM's PagedAttention and continuous batching underneath. This guide covers deploying it on Runpod.

1. One thing to check before you deploy: the license

Aphrodite Engine is licensed AGPL-3.0, unlike vLLM, SGLang, and LMDeploy, which are Apache 2.0. AGPL's network copyleft clause means that if you expose Aphrodite over a network as part of a service, you're obligated to make the source available to users of that service. This doesn't affect using it for internal tooling or personal projects, but it's worth confirming with whoever handles compliance before shipping it inside a commercial product.

2. GPU selection

Aphrodite takes up roughly 90-92% of available VRAM by default, more aggressive than vLLM's typical 0.9 utilization target, so size your GPU with that in mind rather than assuming headroom for other processes on the same card.

RTX 4090 (24 GB): 7B-13B models at EXL2 or GPTQ quantization
L40S (48 GB): 13B at BF16, or 30B-class models quantized
A100 80GB or H100 SXM: 70B at AWQ/GPTQ 4-bit, or BF16 for smaller models with room for LoRA hot-swapping

3. Launching a Runpod Pod with Aphrodite

From Pods > GPU Cloud, deploy on an L40S with:

alpindale/aphrodite-openai:latest

Attach a Network Volume of at least 30 GB mounted at /root/.cache/huggingface.

Under Expose Ports, add port 2242, Aphrodite's default port (a holdover from its KoboldAI origins). Runpod's proxy serves it at https://{POD_ID}-2242.proxy.runpod.net.

Set the start command:

--model meta-llama/Llama-3.1-8B-Instruct --max-model-len 16384

Aphrodite's official examples run with --ipc=host in plain Docker. Runpod doesn't have a separate console field for this; if you're building a custom image and hit shared-memory errors specifically under multi-GPU load, check /dev/shm inside the pod before assuming a code bug.

4. Serving community-quantized checkpoints

This is Aphrodite's actual reason to exist over vLLM. For an EXL2 checkpoint:

--model bullerwins/Meta-Llama-3.1-70B-Instruct-exl2_6.0bpw --quantization exl2 --tensor-parallel-size 2

For GGUF:

--model TheBloke/Llama-2-13B-chat-GGUF --quantization gguf

--tensor-parallel-size must match your GPU count, same convention as vLLM.

5. Limiting VRAM usage

If you're not serving at scale and don't want Aphrodite claiming 90%+ of the card, cap it explicitly:

--model meta-llama/Llama-3.1-8B-Instruct --gpu-memory-utilization 0.6

6. Testing the endpoint

Aphrodite exposes both an OpenAI-compatible API and a KoboldAI-compatible API on the same port. For OpenAI compatibility:

curl https://{POD_ID}-2242.proxy.runpod.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

If you're connecting a KoboldAI-native client like SillyTavern, launch with --launch-kobold-api and point the client at the same port's /api/v1/generate route instead.

7. Modern samplers

The samplers Aphrodite adds over stock vLLM are passed as request parameters, not launch flags: dry_multiplier, xtc_probability, min_p, and mirostat_mode are the ones worth knowing. They matter most for long-running chat and creative-writing workloads where standard temperature/top-p sampling tends toward repetition over long outputs.

8. Monitoring

Aphrodite inherits vLLM's Prometheus metrics on /metrics, plus its own per-LoRA request counters if you're hot-swapping adapters. Existing vLLM Grafana dashboards work with minimal adjustment since most metric names match.

9. Quick-start checklist

Confirm AGPL-3.0 is compatible with how you plan to expose the service
Create a Pod with alpindale/aphrodite-openai:latest, a Network Volume, and port 2242 exposed
Set the start command with --model and, for community quants, --quantization exl2 or --quantization gguf
Cap --gpu-memory-utilization if you don't need Aphrodite's default 90%+ allocation
Test with the curl command above

Deploy on Runpod →