Skip to main contentBlog
Test that your Runpod template's files survive a stop/start or a replacement Pod with a small persistence probe, then set up cache paths, startup, credentials and a release record so you can update and roll back safely.
This is the third article in a series on Runpod templates. If you haven’t read the first and second articles, start there.
Before relying on a Runpod template, check that your files survive the storage lifecycle you plan to use. Then test what happens when you update the application: which code changes, which data stays and what you can restore if the release fails.
This tutorial starts with a small Python script that creates a unique marker and checks it after a stop/start or from another Pod. The later sections cover cache paths, startup, credentials and releases.
Use a disposable Pod with Python 3 and a working web terminal. Select a volume disk or network volume mounted at /workspace, as described below. If you reuse the Part 2 template, change its storage settings first; that template doesn't request persistent storage. The probe needs no extra packages, application API or GPU. For the release section, you'll need an application image and template to update.
Pod runtime and retained storage are billable. Check the deployment estimate before starting.
Three storage types matter for this example:
Both persistent choices normally appear at /workspace; a network volume replaces the volume disk. Runpod's newer Global volumes, currently in beta, are a separate option. See the storage comparison.
A volume disk works for a project you'll resume on the same Pod. Choose a network volume for models or datasets that need to move to replacement Pods. You must select the network volume when creating the Pod; it can't be attached later. Network-volume Pods run in Secure Cloud and are limited to the volume's data center. Check GPU availability there before storing your only working copy on that volume. The network-volume setup guide covers these constraints.
Allow space for downloads and extraction as well as the files you intend to keep. A 20 GB download may coexist with an older model, temporary download files and extracted data. Include outputs, checkpoints and room for the next run in your estimate. Check GPU memory (VRAM) requirements separately from disk capacity.
Storage charges continue after compute stops. A stopped Pod's volume disk bills at twice its running rate, and a retained network volume has its own charges. At zero balance, Runpod's Pod pricing documentation says Pods without network volumes are terminated, with unrecoverable data loss. Network-volume data initially remains, but unpaid storage charges can eventually cause that volume to be terminated too, as the network-volume documentation explains.
Enable balance notifications and keep a separate backup of important files.
Install application code and dependencies in the image, outside the data mount. Paths such as /opt/template-app and /opt/template-venv keep them separate from files retained under /workspace. For a model-based application, use a layout like this:
/opt/template-app/ application supplied by the image
/workspace/template-app/models/ downloaded model revisions
/workspace/template-app/outputs/ generated files and checkpoints
/workspace/template-app/cache/ replaceable cached data
Configure your application's loaders and output paths to use these directories. This is a proposed convention, so the application won't adopt it automatically.
Installing the application under /workspace in the image causes trouble when storage is mounted there: the mount hides the image's files beneath it. Python may report a missing script even though the image contains it. Docker documents the same mount behavior for bind mounts. Arrange any required copying explicitly; don't depend on an empty cloud volume receiving files from the image.
Keep dependencies in the image too, unless you've designed and versioned an editable environment. A persistent virtual environment can retain older packages across image updates and defeat the new dependency lock file.
Before using the data path, inspect it in the Pod terminal:
mountpoint /workspace
df -h /workspace
id
ls -ld /workspace
Confirm the volume selection and mount path in the console as well. /workspace can exist as an ordinary directory without persistent storage behind it. Resolve a failed mount check before writing valuable files, and save important container-disk data before stops, restarts or Pod edits.
Open the Pod's Connect tab. Turn on Enable web terminal, wait for its status to show Running, then click Open web terminal. The web terminal is separate from the HTTP application and doesn't require the image to launch SSH or Jupyter. Runpod describes the controls in its connection guide.
Paste this entire block into the terminal. It writes the standalone script to /tmp, then runs it once to create the marker.
cat > /tmp/persistence_probe.py <<'PY'
"""Create a unique marker, then check it without repairing or replacing it."""
import argparse
import os
from pathlib import Path
import uuid
def main():
parser = argparse.ArgumentParser(description=__doc__)
commands = parser.add_subparsers(dest="command", required=True)
create = commands.add_parser("create", help="write under an existing directory")
create.add_argument("root", type=Path)
check = commands.add_parser("check", help="read an existing marker; never write")
check.add_argument("directory", type=Path)
check.add_argument("--expect", required=True, type=uuid.UUID)
args = parser.parse_args()
try:
if args.command == "create":
token = str(uuid.uuid4())
directory = args.root.resolve() / ("persistence-probe-" + token)
directory.mkdir()
with (directory / "marker.txt").open(
"x", encoding="utf-8", newline="\n"
) as marker:
marker.write(token + "\n")
marker.flush()
os.fsync(marker.fileno())
print("DIRECTORY=" + str(directory))
print("TOKEN=" + token)
else:
marker = args.directory / "marker.txt"
if marker.read_bytes() != (str(args.expect) + "\n").encode("utf-8"):
parser.exit(1, "FAIL: marker contents do not match the saved token\n")
print("PASS: original marker matches " + str(args.expect))
except OSError as error:
parser.exit(1, "FAIL: " + str(error) + "\n")
if __name__ == "__main__":
main()
PY
python3 /tmp/persistence_probe.py create /workspace
The script creates a unique directory, writes a random token and forces the file to storage with os.fsync. It prints DIRECTORY and TOKEN. Save both values on your own computer, alongside the Pod ID and chosen storage. The token is harmless test data.
Replace the placeholders below with the printed values, keeping the quotes:
python3 /tmp/persistence_probe.py check 'DIRECTORY_FROM_CREATE' --expect 'TOKEN_FROM_CREATE'
The expected response is PASS: original marker matches followed by your token. A missing file or different contents gives FAIL: and a nonzero exit status.
Use check for every subsequent test of this marker. It never recreates the file. Running create again produces a different directory, so it can't tell you whether the original file survived.
For a volume disk, finish other work and preserve anything valuable outside the mount. Stop the test Pod through the console, then start that same Pod using the documented stop/start operations.
Reopen the web terminal and paste the script again; its copy in /tmp is temporary. Run only the check command with your original directory and token. You should receive the same PASS response.
For a network volume, record its ID and data center, then create the marker on Pod A. Deploy test Pod B in the same data center with the same network volume mounted at /workspace. Copy the script into B's terminal and check A's saved values. A matching token shows that another Pod can read the retained file. This exercise leaves both Pods in place and doesn't test termination.
Keep the overlap brief because two running Pods cost more than one. The probe on Pod B is read-only, but other processes may still write to shared files. Runpod's network-volume documentation warns that simultaneous writes from multiple Pods can corrupt data. Coordinate application writers before sharing the volume.
Record the before/after Pod IDs, volume identity, command and result. A local run tests the script; the Pod exercise tests the selected cloud storage. The marker confirms that one file survived with its contents intact. Check model files and backup readability separately.
End the exercise according to your retention plan. Charges may continue for a stopped Pod or a volume with no Pod attached.
Trace how your application loads models and saves output before adding path variables to a template. The application or one of its libraries must read each variable. For example, OUTPUT_DIR has no effect on an API that doesn't use it to save responses.
For an application using huggingface_hub, set the documented hub cache variable:
HF_HUB_CACHE=/workspace/template-app/cache/huggingface/hub
Set it before the application imports the library. If the loader also passes a cache argument, check which setting takes effect. Hugging Face documents its cache environment variables. Configure other library caches separately as needed.
Pin model downloads to a reviewed revision, ideally a full commit hash, and keep the repository ID in your release record. Hugging Face's download API accepts explicit revisions and returns a local path. Pass that path to your application's loader. Include any required configuration and tokenizer files, or a separate base model when using an adapter.
A small, fixed model can be baked into the image. Large weights or models that change frequently often fit better on persistent storage, where several application releases can reuse them. Test either arrangement by loading the model and running an inference that reflects its intended use.
Store user outputs separately from caches so cache cleanup leaves generated files and checkpoints intact.
Startup should leave the environment usable each time it runs, including after an interruption. With the layout above, begin by checking the mount and creating the data directories:
if ! mountpoint -q /workspace; then
echo 'Expected persistent mount at /workspace' >&2
exit 1
fi
mkdir -p /workspace/template-app/models \
/workspace/template-app/outputs \
/workspace/template-app/cache
Put the application's foreground launch command after this preparation in your startup script.
For larger assets, check the exact requested revision before downloading and reuse a verified, complete artifact. A directory left by an interrupted download needs further validation.
If you write a custom downloader, use a unique temporary file on the same filesystem as the destination. Verify the expected size and trusted checksum, then atomically rename it into place. Write a completion record after every required file validates. This lets a retry distinguish incomplete downloads from usable files.
When several Pods may initialize the same directory, use a download mechanism that coordinates writers, or populate the directory once before starting readers. A hand-written existence check leaves a race between checking and writing.
Build dependency changes into the next image instead of running an unconditional pip install --upgrade at startup. Log the revision being prepared and its destination. On failure, exit with a useful error message rather than retrying indefinitely.
Use the credential variable your application expects. If it reads APP_TOKEN, for example, add that variable to the template and use its secret selector to choose a stored token. The selector formats the reference for you, as described in Runpod's secrets guide. Create the credential according to the application's requirements. The standalone storage probe needs none.
For model downloads, use a separate token with only the permissions required. The application receives the resolved credential, so give it only to images and code you trust. Configure application authentication separately from HTTPS at the proxy.
Keep credentials out of startup logs, including logs produced by printing the environment. For private downloads during a build, use Docker build secrets instead of embedding tokens in ARG, ENV or copied files.
Check where Hugging Face saves credentials in your image. HF_HOME controls the location of saved credentials as well as cached data; HF_HUB_CACHE changes only the model-cache path. Runpod's current base Dockerfile sets HF_HOME to /workspace/.cache/huggingface/, placing saved tokens on persistent storage by default in images built from it.
Inspect inherited HF_HOME and HF_TOKEN_PATH values and keep token files outside shared storage. Where appropriate, supply HF_TOKEN through a secret and avoid an interactive login that saves a token into a shared persistent directory.
Runpod resolves secrets when the Pod starts. After rotating a secret, preserve any container-disk data, restart the Pod and confirm it is using the new value.
A template stores launch configuration. To reproduce a release, keep those settings alongside the application source and image. Packages installed in a terminal remain changes to that Pod; editing a template affects future deployments rather than rolling out changes to existing Pods. See template configuration for the image and runtime fields.
Give each tested change a new release tag, such as 0.2.1 after 0.2.0. Record the published digest as well, because tags can be overwritten. On a machine with Docker and registry access, replace the example repository and tag with your release:
docker buildx imagetools inspect YOUR_DOCKERHUB_USERNAME/runpod-template-demo:0.2.1
Save the reported sha256:... digest, repository and platform. The corresponding image reference takes the form repository@sha256:.... A multi-platform index and its platform manifests have different digests, so note which you recorded and verify linux/amd64 support. Docker's digest guide explains the distinction. Retain earlier registry artifacts for rollback.
Create a separate private candidate template with the new release tag in Container image. To use a digest reference in that field, first verify that the current form accepts it and the Pod pulls it successfully.
Copy the intended port, mount, environment settings and secret references. Leave the container start command blank to use this image's own command. Once the candidate passes testing, update the template used for future deployments and deliberately replace or update the running Pods.
Keep the following together in the release record:
When refreshing the base image, review the Runpod PyTorch tags and image README. Recheck compatibility before carrying forward an old tutorial tag.
Test the candidate on a separate Pod using disposable data or a verified copy. Check the health endpoint, if the application has one, and perform a typical operation through its intended interface. Verify the persistence marker. For a model application, test a start with the model already cached and inspect the generated output. Direct clients to the new Pod after those checks pass.
Rollback restores the previous image and compatible configuration, but files changed by the new version remain changed. Before a release migrates data, preserve a recoverable copy and test that the old application can use it.
Check the missing path. /tmp/persistence_probe.py is temporary; paste the script again. Missing image code under /workspace suggests that the mount is hiding it.
If the marker is missing, check the original directory, mount and volume ID before creating anything new.
Inspect the loader's resolved path, requested revision and retained metadata. A cache under /root/.cache is on the container disk unless you've mounted storage there. Investigate those paths before deleting a persistent cache.
Use df -h and df -i to check available space and inodes. Compare id with directory ownership to see whether the application user can write. Adjust permissions on the intended directory; avoid recursively changing an entire shared volume.
The original machine may have no free GPU even though the files are retained. Runpod documents zero-GPU data recovery: click Start and choose CPU Only to reach the Pod's local storage.
The CPU-only probe can inspect files when terminal access is available. An application that requires CUDA may fail to start in this mode, so use the terminal to check storage independently.
Compare the old and new release records, including data formats, model revisions, settings and secrets. Restoring the image alone may leave one of those other changes in place.
Build your next Runpod template with storage you've tested and a release you can reproduce. Use coding agents like Claude Code, Codex and Gemini CLI to help manage your GPU workloads.
Blog Posts
