Building Runpod Templates Part 2: Adding a Python API

This is the second article in a series on Runpod templates. If you haven’t read the first article, start there.

An application template should start the right process and give you a useful way to reach it. For a Python API, that means packaging the code, starting a web server and connecting its listening port to Runpod’s HTTP proxy.

Build a small FastAPI service that accepts a list of numbers, doubles them with PyTorch and returns the result. The service starts automatically, requires a bearer token for calculations and reports which device produced the answer. Test it on your computer’s CPU before taking the same image to a GPU Pod.

Doubling three numbers doesn’t justify renting a GPU. The workload is small on purpose: the answer is obvious, so you can concentrate on packaging, startup, authentication and networking. This is a Pod application with a continuously running web server; it doesn’t implement Runpod’s Serverless worker interface.

You’ll need Docker running Linux containers, Python 3.10 or newer on your computer to generate tokens, a registry account and a Runpod account for the later GPU check. The commands below use Bash, including Bash inside WSL on Windows.

Decide what the application promises

The API has two routes. Public GET /healthz returns only {"status":"ok"} once startup succeeds. Authenticated POST /double accepts {"numbers":[1,2,3]} and returns the doubled numbers, device and template label. It writes no results to disk.

Three environment variables configure it: APP_DEVICE is cpu locally or cuda on the Pod; APP_TOKEN is a private, randomly generated credential; and TEMPLATE_LABEL helps identify the configuration serving your request. These settings take effect because the application’s Python code reads them from the environment.

The server listens on port 8000 with one worker. Its code lives in /opt/template-app, and its Python environment lives in /opt/template-venv. Keeping both outside /workspace prevents a data volume mounted there from hiding the application or its dependencies.

Choose the base and add dependencies

Choose a compatible Runpod PyTorch base from the Docker Hub tags and the official PyTorch image README. Check that it supports linux/amd64, uses Python 3.10 or newer and supports the intended GPU architecture and host CUDA capability. A tag that appears in an older example isn’t a recommendation; pick a current release.

Make a working directory named runpod-template-app-example. Put this in requirements.txt:‍


fastapi==0.162.0
pydantic==2.14.1
uvicorn==0.54.0

These direct dependency pins were checked against the FastAPI, Pydantic and Uvicorn release records on October 28, 2026. All require Python 3.10 or newer. They pin direct dependencies only, not every transitive dependency or base-image package. Recheck them before publishing your own image.

There’s no torch entry because the base already supplies it. The virtual environment uses --system-site-packages, which makes the base interpreter’s installed packages visible. That saves reinstalling the GPU stack, but also means this environment shares packages with the base. The Dockerfile checks that PyTorch still resolves to the same file after installation. See Python’s virtual environment documentation.

Write the service

Create app.py:‍

import os
import re
import secrets
from contextlib import asynccontextmanager
from typing import Annotated

import torch
from fastapi import Depends, FastAPI, HTTPException, Request
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from pydantic import BaseModel, ConfigDict, Field


@asynccontextmanager
async def lifespan(app: FastAPI):
    token = os.environ.get("APP_TOKEN", "")
    if not re.fullmatch(r"[A-Za-z0-9_-]{32,128}", token):
        raise RuntimeError("APP_TOKEN must be 32-128 URL-safe letters/digits/_/-.")
    device_name = os.environ.get("APP_DEVICE", "cuda")
    if device_name not in {"cpu", "cuda"}:
        raise RuntimeError("APP_DEVICE must be cpu or cuda.")
    if device_name == "cuda" and not torch.cuda.is_available():
        raise RuntimeError("CUDA unavailable; check the Pod GPU and image compatibility.")

    device = torch.device("cuda:0" if device_name == "cuda" else "cpu")
    probe = torch.tensor([1.0], dtype=torch.float32, device=device) * 2
    if probe.cpu().tolist() != [2.0]:
        raise RuntimeError("Startup tensor check failed.")
    app.state.token = token
    app.state.device = device
    app.state.label = os.environ.get("TEMPLATE_LABEL", "template-app")
    yield


app = FastAPI(lifespan=lifespan, docs_url=None, redoc_url=None, openapi_url=None)
security = HTTPBearer(auto_error=False)


async def authorize(
    request: Request,
    credentials: Annotated[HTTPAuthorizationCredentials | None, Depends(security)],
):
    if credentials is None or not secrets.compare_digest(
        credentials.credentials.encode("utf-8"), request.app.state.token.encode("utf-8")
    ):
        raise HTTPException(
            status_code=401,
            detail="Invalid or missing bearer token",
            headers={"WWW-Authenticate": "Bearer"},
        )


Number = Annotated[
    float, Field(strict=True, ge=-1_000_000, le=1_000_000, allow_inf_nan=False)
]


class DoubleInput(BaseModel):
    model_config = ConfigDict(extra="forbid")
    numbers: list[Number] = Field(min_length=1, max_length=1024)


@app.exception_handler(RequestValidationError)
async def invalid_input(request: Request, exc: RequestValidationError):
    # Don't echo rejected input, including non-finite numbers, into JSON responses.
    return JSONResponse(status_code=422, content={"detail": "Invalid numbers payload"})


@app.get("/healthz")
async def healthz():
    return {"status": "ok"}


@app.post("/double", dependencies=[Depends(authorize)])
async def double(payload: DoubleInput, request: Request):
    # shortcut: this tiny operation blocks the event loop; use a job queue for long work.
    with torch.inference_mode():
        result = torch.tensor(
            payload.numbers, dtype=torch.float32, device=request.app.state.device
        ) * 2
    return {
        "doubled": result.cpu().tolist(),
        "device": str(result.device),
        "label": request.app.state.label,
    }

‍


FastAPI’s lifespan hook runs the startup function before the server accepts any request. A missing token produces a clear error. Asking for CUDA on a machine where PyTorch can’t find it also fails. There’s no automatic CPU fallback to make a GPU deployment look successful.

Startup then runs a real tensor operation, because a device can be detected while a later operation fails on an incompatible build. Copying the probe result back to the CPU makes startup wait for that calculation to finish.

The input model accepts between 1 and 1,024 finite numbers, each from −1,000,000 to 1,000,000. Strings, booleans, extra fields and out-of-range values are rejected. The generic validation response also avoids trying to serialize rejected NaN or infinity values back into JSON. Pydantic documents the numeric and collection constraints used here.

Calculations use 32-bit floats, so arbitrary decimal inputs can be rounded; the integer example’s answer is exact.

The health route reveals no device information or configuration; those appear only in an authenticated calculation response. Automatic API documentation routes are disabled to keep this example’s public interface small.

These limits bound the tensor workload after JSON parsing; they don’t limit request-body size or request rate. A service intended for untrusted traffic needs those controls, monitoring and a more deliberate credential lifecycle. Long calculations also need a different execution design: this tiny operation runs directly in the event loop and would delay other requests if expanded into a slow job. Requests through Runpod’s HTTP proxy also time out after 100 seconds with a 524 error, so slow jobs need a submit-and-poll design.

Package it with an explicit startup command

Create Dockerfile:

ARG BASE_IMAGE
FROM ${BASE_IMAGE}

WORKDIR /opt/template-app
ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 APP_DEVICE=cuda

COPY requirements.txt ./requirements.txt
# Reuse the base's torch; fail the build if the new environment can't see it unchanged.
RUN python -c "import torch; print(torch.__file__)" > /tmp/template-base-torch.txt \
    && python -m venv --system-site-packages /opt/template-venv \
    && /opt/template-venv/bin/python -m pip install --no-cache-dir -r requirements.txt \
    && /opt/template-venv/bin/python -c "import pathlib, torch; assert torch.__file__ == pathlib.Path('/tmp/template-base-torch.txt').read_text().strip(), 'Base torch was hidden or replaced'" \
    && rm /tmp/template-base-torch.txt

COPY app.py ./app.py
EXPOSE 8000

# Start the API in place of the base's services and health probe.
HEALTHCHECK NONE
ENTRYPOINT []
CMD ["/opt/template-venv/bin/python", "-m", "uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000", "--workers", "1"]

And .dockerignore:‍


*
!Dockerfile
!requirements.txt
!app.py

Installing dependencies before copying app.py lets Docker reuse that installation layer when only application code changes. The allowlist keeps local credentials, virtual environments and unrelated files out of the build context. There is no token in a build argument, image environment variable or copied file.

This Dockerfile replaces the base image’s startup: ENTRYPOINT [] clears any inherited entrypoint, and the exec-form CMD launches the virtual environment’s Python as the foreground process. An inherited health check is disabled because it may probe a service that no longer starts. The HTTP checks below test the API’s own routes.

Changing only CMD can leave an inherited entrypoint involved in startup. Clearing both makes this image’s behavior explicit, following Runpod’s application-only template example and Docker’s command and entrypoint rules. As a result, this image doesn’t start SSH or Jupyter. Preserving those services would require a different startup arrangement and separate tests.

Uvicorn binds to 0.0.0.0 so traffic arriving through the container network can reach it. One worker means one application process; adding workers would create more processes and, for a real model, potentially more model copies. Development reload stays disabled. The relevant options are in Uvicorn’s settings reference.

Build and inspect version 0.2.0

In your working directory, run docker buildx imagetools inspect runpod/pytorch:REPLACE_WITH_TAG for your chosen base and record its top-level digest. The digest identifies the image content even if the tag later changes. Replace the tag, the hex characters after sha256: and your Docker Hub username below:‍

export BASE_IMAGE='runpod/pytorch:REPLACE_WITH_TAG@sha256:REPLACE_WITH_DIGEST'
export IMAGE='YOUR_DOCKERHUB_USERNAME/runpod-template-demo:0.2.0'

docker pull --platform linux/amd64 "$BASE_IMAGE"
docker image inspect "$BASE_IMAGE" --format 'ENTRYPOINT={{json .Config.Entrypoint}} CMD={{json .Config.Cmd}}'
docker run --rm --platform linux/amd64 --entrypoint python "$BASE_IMAGE" -c 'import sys, torch; print(sys.version); print(torch.__version__, torch.__file__)'

docker build --platform linux/amd64 --build-arg BASE_IMAGE="$BASE_IMAGE" -t "$IMAGE" .
docker image inspect "$IMAGE" --format 'ENTRYPOINT={{json .Config.Entrypoint}} CMD={{json .Config.Cmd}}'

‍
The first inspection shows what you’re replacing. The Python command checks the base interpreter and PyTorch import without running its usual startup. The final inspection should show an empty entrypoint and the Uvicorn command from the Dockerfile. Keep this output with your build records.

An ARM computer needs Docker support for building and running linux/amd64, or a suitable remote build machine. The first download can be large. If the environment check fails, inspect the selected base’s Python setup; don’t fix it by blindly reinstalling PyTorch or copying a whole environment into /workspace.

Start it locally and test the HTTP interface

Generate a token in your current terminal and launch the image:‍

export APP_TOKEN="$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))')"

docker run -d --name template-app-local --platform linux/amd64 \
  -p 127.0.0.1:8000:8000 \
  -e APP_DEVICE=cpu -e APP_TOKEN -e TEMPLATE_LABEL=local-app \
  "$IMAGE"
docker logs -f template-app-local

‍Wait for Uvicorn’s application-startup message. Press Ctrl+C to stop following logs; the detached container stays running. Binding the host side to 127.0.0.1 keeps the test API reachable only from your computer; inside the container, Uvicorn still needs 0.0.0.0.

In the same terminal, check both routes:‍

curl --fail-with-body  http://127.0.0.1:8000/healthz
curl --fail-with-body http://127.0.0.1:8000/double \
  -H "Authorization: Bearer $APP_TOKEN" \
  -H 'Content-Type: application/json' \
  --data '{"numbers":[1,2,3]}'

‍The expected calculation response is:

{"doubled":[2.0,4.0,6.0],"device":"cpu","label":"local-app"}

Now check the failure paths. A valid payload without authorization must get 401; an empty list with valid authorization must get 422:‍

curl -i http://127.0.0.1:8000/double \
  -H 'Content-Type: application/json' --data '{"numbers":[1]}'
curl -i http://127.0.0.1:8000/double \
  -H "Authorization: Bearer $APP_TOKEN" \
  -H 'Content-Type: application/json' --data '{"numbers":[]}'

python3 smoke_test.py http://127.0.0.1:8000 \
  --expect-device cpu --expect-label local-app

‍Restart the container with docker restart template-app-local, wait for startup again and rerun the requests above. This catches assumptions that only held on the first launch. These CPU tests cover the image and its HTTP behavior; GPU compatibility requires the Pod calculation below.

Push the image and configure the Pod template

Create or reuse your Docker Hub repository, then publish the version you just checked:‍

docker login
docker push "$IMAGE"
docker buildx imagetools inspect "$IMAGE"

‍Record the pushed digest along with the tag; tags can be moved later, so the digest is the stronger identity record. This example uses 0.2.0 as its release tag; choose an unused version in your own repository rather than overwriting an earlier release. Private registries also require Runpod pull credentials.

Create a private Pod template in the console. Use these settings, checking the current template fields:

• Name: template-app-0.2.0
• Image: YOUR_DOCKERHUB_USERNAME/runpod-template-demo:0.2.0
• Compute: NVIDIA GPU compatible with the base’s CUDA and GPU architecture requirements
• Container start command: Leave blank so the image’s Uvicorn command runs
• HTTP ports: 8000
• TCP ports: None
• Container disk: Enough for the unpacked image plus runtime headroom; check the image size and deployment estimate
• Volume disk: 0 GB if allowed; otherwise the smallest offered size, mounted at /workspace
• APP_DEVICE: cuda
• TEMPLATE_LABEL: pod-app
• APP_TOKEN: A Runpod secret reference, as described next

The application needs no persistent storage, so any volume you select sits unused. You can inspect image size in bytes with docker image inspect "$IMAGE" --format '{{.Size}}'; don’t size disk using only the compressed download figure.

Generate a fresh token for the Pod with python3 -c 'import secrets; print(secrets.token_urlsafe(32))' and store it privately as a Runpod secret, following the name requirements shown in the console. In the template, add an environment variable named APP_TOKEN, then click its secret selector to choose the stored token. The selector formats the reference automatically as {{ RUNPOD_SECRET_your_secret_name }}. Runpod resolves it when the Pod starts; the token isn’t baked into the image. See Runpod’s secrets instructions.

Port exposure is separate from starting a process. EXPOSE 8000 documents the image’s port; the HTTP-port setting enables Runpod’s proxy route; Uvicorn is the process that listens. The proxy provides HTTPS, but its URL is public and doesn’t replace application authentication. See Runpod’s port guide.

Verify the GPU response through Runpod

Deploy from the saved template, selecting an available compatible GPU and a host CUDA capability meeting the base image’s requirement. Review compute and storage charges before deploying. Watch the Pod logs until application startup completes, then copy the port-8000 HTTPS URL from Connect. Use that URL rather than constructing one from memory.

In your local Bash terminal, enter the same token you stored in Runpod:‍

read -r -s -p 'Pod APP_TOKEN: ' APP_TOKEN
export APP_TOKEN
printf '\n'
export POD_URL='https://REPLACE_WITH_YOUR_PORT_8000_PROXY_HOST'

curl --fail-with-body "$POD_URL/healthz"
python3 smoke_test.py "$POD_URL" --expect-device cuda:0 --expect-label pod-app

‍The successful calculation now reports "device":"cuda:0" and "label":"pod-app", with the same doubled numbers. The device comes from the tensor used for the answer. This verifies an external request, authentication and an actual CUDA operation together; the Pod’s Running label alone can’t establish that.

When a step fails

The process exits during startup

Read the application traceback in container or Pod logs. Check that the secret resolves to a 32–128-character URL-safe token and APP_DEVICE is exactly cpu or cuda. A literal unresolved secret reference deliberately fails token validation. Recreate the local container after changing its environment; exporting a new value doesn’t change an existing container.

The proxy returns 502

Confirm startup completed, the saved template exposes HTTP 8000 and the command binds 0.0.0.0:8000. Check for an accidental container start-command override. Runpod’s 502 troubleshooting guide walks through process and port checks.

Health works, but calculations return 401 or 422

Networking is working. Check the bearer token for 401, or the request shape and numeric limits for 422. Opening /double in a browser address bar sends GET, while this route requires POST.

CUDA startup fails

Check the GPU attachment, image CUDA requirements, host driver capability and supported GPU architecture. Don’t set APP_DEVICE=cpu and count the result as a GPU pass. A successful build only proves what happened on the build machine.

Finish the test and keep the release reproducible

Remove the local test container when finished:

docker stop --time 10 template-app-local
docker rm template-app-local
unset APP_TOKEN

Save any needed Pod logs, then terminate the disposable Pod and verify that it is gone. Stopping compute can leave storage charges; consult the current Pod pricing. The image and template remain available for your next deployment.

For an application change, edit the source, build a new image version and repeat the requests before updating the template. Keep the tested image digest and launch settings with your source files so a fresh deployment starts the version you intended.