How to Run OpenAI Agents SDK on Runpod

The OpenAI Agents SDK is provider-agnostic despite the name: it supports the OpenAI Responses and Chat Completions APIs natively, and reaches 100+ other model providers through a LiteLLM adapter. That means you can build an agent with the SDK's tools, guardrails, and handoffs, and point the actual model calls at a vLLM or TGI endpoint you're already running on Runpod instead of paying OpenAI per token. This guide covers that setup.

1. The two layers: agent orchestration and model inference

Same pattern as any agent framework on Runpod: the orchestration logic (the Agents SDK's Python runtime, deciding which tool to call and when) is cheap and CPU-bound. The actual model inference is GPU-bound and expensive. Splitting them into two Runpod deployments, a CPU pod for the agent and a Runpod Serverless endpoint for the model, lets each scale independently instead of paying for idle GPU time whenever the agent is just waiting on a tool call.

2. Deploy the model layer first

Deploy a vLLM worker from the Runpod Hub as a Serverless endpoint, or a TGI pod, following the same GPU sizing guidance as any inference deployment: an L40S for a 7B-13B model is a reasonable default. Once it's live, you have an OpenAI-compatible endpoint at either a Pod's proxy URL or a Serverless endpoint's https://api.runpod.ai/v2/{ENDPOINT_ID}/openai/v1 path.

3. Install the SDK with LiteLLM support

pip install "openai-agents[litellm]"

The SDK includes a built-in LitellmModel that routes through LiteLLM without requiring a separate LiteLLM proxy process, useful when you just want to point at one self-hosted endpoint without standing up a full gateway.

4. Point an agent at your Runpod-hosted model

from agents import Agent, Runner, function_tool
from agents.extensions.models.litellm_model import LitellmModel

@function_tool
def get_weather(city: str) -> str:
    """Look up the weather for a city."""
    return f"The weather in {city} is sunny."

model = LitellmModel(
    model="openai/meta-llama/Meta-Llama-3.1-8B-Instruct",
    base_url="https://{POD_ID}-8000.proxy.runpod.net/v1",
    api_key="placeholder",
)

agent = Agent(
    name="weather_agent",
    instructions="You are a helpful weather assistant.",
    model=model,
    tools=[get_weather],
)

result = Runner.run_sync(agent, "What's the weather in Austin?")
print(result.final_output)

The openai/ prefix in the model string tells LiteLLM to treat your Runpod endpoint as an OpenAI-compatible API rather than route it through a specific provider's SDK, since vLLM and TGI both speak that API already.

5. Routing across multiple models

If different agents in your system should hit different backends (a fast, cheap self-hosted model for routing decisions, a larger self-hosted or cloud model for the final response), define a custom ModelProvider that resolves model names to different LitellmModel instances, and pass it via RunConfig(model_provider=your_provider) when you call Runner.run. This is also the pattern to use if you're routing through a full LiteLLM proxy deployment rather than the SDK's built-in adapter, pointing the provider's client at your proxy's base URL and letting the proxy's config handle the per-model routing.

6. Deploying the agent layer on Runpod

Package your agent code (the Python runtime, tool definitions, and a handler) in a Dockerfile, expose it as a Runpod Serverless CPU endpoint using the standard handler pattern, or run it on a lightweight Pod if you need it always-on rather than scale-to-zero. Nothing about the agent orchestration layer needs a GPU; provisioning one for it is wasted spend.

7. Tracing

The SDK's default tracing assumes OpenAI's platform is receiving trace data. When you're not using OpenAI models at all, disable it explicitly:

from agents import set_tracing_disabled
set_tracing_disabled(disabled=True)

Otherwise the SDK will attempt to send trace data to an endpoint you're not using.

8. Common integration issue

If LiteLLM emits Pydantic serializer warnings for response objects when parsing your self-hosted model's output, the SDK ships a compatibility patch:

export OPENAI_AGENTS_ENABLE_LITELLM_SERIALIZER_PATCH=true

It's disabled by default because it depends on a private LiteLLM API; re-validate it after upgrading LiteLLM versions.

9. Quick-start checklist

Deploy your model on a Runpod Pod or Serverless endpoint with an OpenAI-compatible API (vLLM or TGI)
pip install "openai-agents[litellm]"
Build an agent using LitellmModel pointed at your Runpod endpoint with the openai/ prefix
Disable tracing if you're not using OpenAI's platform
Deploy the agent code itself on a CPU pod or Serverless endpoint, separate from the GPU layer

Deploy on Runpod →