How Runpod’s agent skills and MCP server work together

Agents are a growing share of Runpod’s users, according to the State of AI Compute. I had spent the summer on the agent side of Runpod, and I kept seeing the same thing: agents were already running infrastructure for our customers, with only the API to go on. We built skills to teach them how Runpod works.

This is the story of how we landed on skills and why they matter, where skills fell short and what we built underneath to keep them right. Setup takes one prompt, and everything else we shipped this month has its own post.

What an agent gets wrong with just the API

Give a coding agent the Runpod tools and ask it to serve a model. Here is what I watched happen, over and over:

• It builds a custom image for a model that already has a maintained worker on the Hub, then fights with linux/amd64, registry credentials and a cold start it did not expect.
• It creates a Pod with the GPU you named, in whatever data center it lands in, and then cannot attach your network volume, because a volume lives in one data center and the Pod went to another.
• A stopped Pod will not start again, the data center shows plenty of stock, and the agent tells you capacity is the problem. It is not: a Pod is bound to the machine it was first placed on, and that machine's GPUs went to someone else while the Pod was stopped (Zero GPU Pods on restart).
• A Serverless job fails with "job timed out after 1 retries", and the agent recommends more workers.

None of these is a model problem. Every one of them is a missing piece of knowledge about how Runpod works, and the API can’t carry that knowledge. A tool call tells the agent what it can do. It never tells it whether it should, or in what order.

Why we built skills

So the agent product became two things. The MCP server is the hands: every control-plane operation as a typed tool call. Skills are the knowledge: Markdown the agent reads before it acts (skills docs). They tell the agent which tool to reach for (the MCP server for infrastructure, runpodctl for a terminal or CI, Flash for deploying your own Python), the procedure for each kind of job, end-to-end recipes and the rules of how Runpod works. The MCP server is hosted at mcp.runpod.io.

The recipes were run, not written. In July I ran whole tasks on a real Runpod account, the way a user would ask for them: stand up Ollama on a Pod and give me the URL, deploy Whisper as an endpoint, fine-tune on a Pod and serve the result, serve the same handler from three regions. Twenty-five recipes, most of them run live, with every command and output kept and every resource torn down after. Each run caught a gap in the skills or something the docs did not say, and the fix went back into the skills.

Here is what that does to a failed Serverless job. The status payload says "job timed out after 1 retries" and names a worker id. The skill tells the agent not to trust the error string: the worker id is the handle, and the worker's own log names the cause (Monitor logs).

runpodctl serverless logs <endpoint-id> --worker z3jr9rv2k3tt2y --source container Failed to get job. | Error Type: Exception | Error Message: Job has missing field(s): id or input.

The worker was healthy and rejecting an empty payload. That is the answer you get, not a guess about capacity. And the Pod-and-volume mistake never happens, because a rule is read before the first call: a network volume attaches only to a Pod in the volume's data center, so set dataCenterIds to the volume's dataCenter, or leave it empty.

We tested this on a prompt that asks for one endpoint with the same weights in three data centers. Two runs had one line in the skills, read the worked example before calling tools, and two runs did not, with the same model and the same tools. Without that line, both runs invented a limitation to justify their own plan: "you cannot attach one network volume to a multi-DC endpoint", which the July run had already done. With it, both opened the recipe before planning and attached one volume per data center to one endpoint. One sentence of knowledge was the difference.

Where agent skills fell short

The first version of the skills was me writing down what I knew. It worked for a few weeks, and then it started lying, in small ways. runpodctl shipped serverless health in one release and pod logs in the next, and the skills still said the CLI could not do either: eleven statements, including an eval that graded an agent correct for saying so. Nobody noticed, because nobody re-reads a document that was right last month. An agent reading "this tool cannot do X" has no reason to check, so it does not.

That is the shortcoming of skills as static knowledge. They are text, and text drifts away from the thing it describes. A rule buried in a paragraph has no source to check it against, no date and no test that fails when the product changes under it. A skill without a test is a rumor. So most of the work since August has gone into the layer underneath: knowledge that is sourced, checked and tested, and that the skills draw on.

How we added knowledge

Every rule comes from a source, not from memory. The facts about how Runpod works now live in a concept graph: Pods, endpoints, volumes, GPU types and workers, each with its fields, states and rules. This is the rule behind the stopped-Pod case above, as it ships:

- id: pod.stopped-pod-keeps-host  statement: >    A stopped pod stays tied to its machine and resumes on the same host. If another user has taken the GPU, you can wait, start the pod with zero GPUs to copy data off, or migrate it to a new machine. Keep important data on a network volume so you can redeploy on any machine.  evidence:    - { source: rest-v2-spec, ref: Pod.cudaVersion, seen: "2026-09-28" }    - { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/zero-gpus" }

Every rule looks like that: one testable sentence, its evidence and the date it was checked when the fact can change. The evidence is the public docs, the API reference or a probe we ran when no doc had the answer; the rule that says /workspace is a network filesystem in some data centers and a local disk in others came from measuring it. The build rejects a rule whose evidence a reader cannot find, and a reviewer signs off before it ships. The next version of the graph is not typed by hand either: an agent writes it from a written brief and Runpod's own sources, it is checked and reviewed before it ships, and when two sources disagree the rule keeps both sides and says so.

The same knowledge reaches every agent. Every skill and recipe is linked to the concepts it relies on, and the graph is also served from inside the MCP server (MCP docs), so an agent with only the MCP connected can ask what binds a Pod to its machine and get the rule above, with its source.

Every claim is checked on every change. A rule without evidence, a link that does not resolve, a guide that belongs to no concept or a "this tool cannot do X" statement that the current CLI contradicts all fail the build. Those eleven stale claims cannot ship now.

Behavior is tested, not read. Each skill ships eval cases, more than 80 across the plugin, each a prompt, the expected behavior and the assertions. One of them pins the split between a one-line call ("stop Pod abc123") and a multi-step job, and checks that the agent consults the tested path instead of presenting a hand-derived plan.

The whole thing runs against a live account. This is the part I would keep if I had to drop everything else. We keep a catalog of 52 user stories, from "which image gen GPU is available and what it costs" to "ComfyUI web UI returns 502", plus two traps where the right answer is to change nothing. The stories come from what people actually ask: usage data proposes which to add or re-aim, and a person decides. Each story is run by a real agent, connected to the MCP and the skills, against a real Runpod account, with prompts phrased the way people type them, no ids pasted in. Two graders score every run. One checks the recorded tool calls, their order and arguments. The other sees only the agent's final answer and checks that it says true, specific things. Every story runs twice, with the skills and without them, on the same server and the same model, so we can measure the lift the skills give. When a story fails, a repair step proposes one edit to one skill, and that edit goes through the same pipeline as everything else.

Agents report back. Agents can report a rule that was wrong or missing, or a question they could not answer. Those reports are grouped by concept so a person reviews clusters instead of single reports, and the next build re-checks each flagged concept against its sources before any rule changes.

What the loop produced in November

The newest change to the skills went out in the first week of November. The skill for deploying Pods had agents create a network volume first and the Pod second, and on a stock refusal it offered another data center, which with a volume means deleting it and creating a new one each time. The change added one hard rule to the skill: create a network volume only after the stock read has chosen its data center, only one per journey unless the user chooses to move, and never delete the volume to try another data center.

The rule in the concept graph, the recipe steps and the eval that pins the behavior shipped in the same change. That is what the system is for. A fix is not a sentence someone remembered to add; it ships as a rule with evidence, a test that fails without it and a live story that proves it.

Why this matters

The skills are what separate an agent that can call the Runpod API from one you would let touch your account. The model is not the difference: the same model, with the same tools, deploys the wrong image, loses the volume and misreads the timeout, until it has the knowledge. And knowledge that is not tested rots in weeks.

So the thing I built is not really a set of documents. It is a loop: sources become rules, rules become skills, skills are run against a live account by real agents, failures become edits and agents in the field report what the loop missed. The documents are just what the loop produces this week.

Try it

Paste this into your coding agent:

Read agents.runpod.io and set up Runpod for me.

Then give it a real job, like serving a model or finding out why your last Serverless job failed, and watch what it checks before it acts. If it gets something wrong, it can report that itself, and that report is where the next fix starts.

Next steps

Prefer to install it yourself? The agent setup docs have the commands for each client and for CI.

Want to review your agent’s changes as a Terraform plan before anything bills? Read Manage Runpod as code with Terraform provider 2.0.

Want everything else we shipped this month? Read From setup to deployment: what’s new in Runpod for Agents.