Managed APIs send your data to someone else's model on someone else's servers. For most use cases, that's fine. For agents processing medical records, legal documents, proprietary customer data, or anything subject to HIPAA, GDPR, or internal data governance policies, it's not.
Self-hosting means every part of your agent stack runs on infrastructure you control: your models, your inference server, your data. This guide covers what that looks like in practice, where Runpod fits in, and how to avoid the failure modes that trip up most private agent deployments.
What Does Self-Hosting AI Agents Actually Mean?
A self-hosted agent system has three components, each of which can be private or managed independently:
The LLM inference layer is the one that matters most for data privacy. When your agent passes tool results (database contents, document excerpts, user data) to the model for reasoning, that data goes wherever the model is running. If the model is on OpenAI's servers, so is your data.
Self-hosting the LLM layer means running an open-source model (Llama 3, Mistral, Qwen) on GPU infrastructure you control.
Why Use Runpod for Self-Hosted AI Agents?
Runpod gives you raw GPU compute, the infrastructure layer that open-source LLMs actually run on. You bring the model. You bring the inference server. You control the full stack.
This is different from most self-hosted AI offerings that still abstract away the model or the inference configuration. On Runpod, you're running your own Docker container, with your own vLLM flags, on your own GPU, inside your own account.
- Secure Cloud: Runpod-owned T3/T4 data center hardware, enterprise-grade isolation. Infrastructure partners hold SOC 2, ISO 27001, and PCI DSS certifications. For agents running in Secure Cloud, you're not sharing hardware with unknown community GPU providers.
- Persistent encrypted volumes: model weights and any data cached to disk stay inside your Runpod account, encrypted at rest. Container and volume disks are billed per second; network volumes (shareable across pods) are billed hourly.
How Do I Choose a Model for My Private AI Agent?
You're running an open-source model. The right choice depends on your VRAM budget and the reasoning demands of your agent tasks. A few examples:
4-bit quantization (GPTQ or AWQ) reduces VRAM to roughly 25–30% of the FP16 footprint — a 65–70% reduction — with limited quality degradation for most agent reasoning tasks. The table above reflects these figures: an 8B model that requires ~16GB at FP16 drops to ~5GB at 4-bit; a 70B model drops from ~140GB to ~40GB.
It's typically the right default for self-hosted deployments unless you have specific accuracy requirements that demand full precision.
How Do I Deploy a vLLM on a Secure Cloud?
The full deployment walkthrough — Runpod Hub template, custom Docker container, and framework connection — is covered in the Deploying and Hosting AI Agents guide. The only step that changes here is your deployment target.
When creating your pod, select Secure Cloud rather than Community Cloud. Community Cloud hardware is contributed by independent providers; Runpod cannot guarantee the same physical and network isolation.
Secure Cloud runs on Runpod-owned data centre hardware with enterprise-grade isolation; your model weights, input data, and inference outputs never leave Runpod-controlled infrastructure.
Running AI Agents Locally: Setup, Limitations, and When to Use Cloud
For fully air-gapped environments or development without a cloud account, you can run the full stack locally. The constraint is GPU hardware.
Setup with Ollama (simplest path)
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull a model
ollama pull llama3:8b
# Start the server (OpenAI-compatible endpoint at localhost:11434)
OLLAMA_HOST=0.0.0.0 ollama serve
# Point your agent framework at it
# base_url = 'http://localhost:11434/v1'
Ollama is the right choice for local development and testing. For production — where you need concurrent request handling, consistent performance, and proper resource management — vLLM on a Runpod Secure Cloud Pod is the better path.
Consumer GPU limitations
An RTX 4090 (24GB VRAM) runs 7B–13B models at FP16 or 13B–34B models with 4-bit quantization. Running a 70B model requires a higher-VRAM GPU — at 4-bit, a 70B model still needs approximately 35–40GB of VRAM, which exceeds the 4090’s 24GB. For 70B models, use an A100 80GB or a multi-GPU Runpod pod. Below 24GB VRAM, you’re limited to smaller models or heavy quantization. If you need larger models or lower latency than local hardware can provide, a cloud GPU instance on Runpod extends your local setup on-demand — without the capital expense of buying server-grade hardware.
How Runpod Hub Simplifies Self-Hosted Agent Deployment
Runpod Hub is a catalog of pre-built, community-contributed container templates for common AI tools. For self-hosted agent deployments, this matters because it lowers the barrier to running complex open-source stacks that would otherwise require significant Docker and configuration work.
Templates in Hub include ready-to-run configurations for vLLM, Ollama, common vector databases, and agent UI frameworks. You deploy them to a Runpod Pod with one click, modify the configuration to suit your setup, and have a running service in minutes.
This is particularly useful for teams that want the privacy of self-hosted infrastructure but don't want to build and maintain all the DevOps work themselves. Runpod handles the compute; Hub handles the software setup; you control the data and the models.
What Are the Major Compliance Considerations for Private AI Agent Deployments?
Compliance for AI agents isn't primarily about where the model runs; it's about what data the agent processes. When an agent passes document contents, user inputs, or tool results to the LLM for reasoning, that data flows through the inference layer. Understanding which regulatory frameworks apply to that data, and what each framework actually requires, determines your architecture before you pick a cloud provider.
GDPR
The General Data Protection Regulation applies whenever you process personal data belonging to EU residents, regardless of where your infrastructure is located. For AI agents, the key obligations are: establishing a lawful basis for processing (Article 6); entering into Data Processing Agreements with any vendor whose infrastructure touches personal data (Article 28); and applying data minimisation — only passing to the agent the subset of data actually needed for the task, not full customer records.
International data transfers require appropriate safeguards under Chapter V. If your inference endpoint is hosted outside the EU, you need either an adequacy decision covering that country or Standard Contractual Clauses in place with your provider. Self-hosted inference on EU-region infrastructure avoids this requirement entirely.
If agent interactions are logged, the right to erasure (Article 17) applies to any personal data in those logs. Most inference servers, including vLLM, do not log prompts by default; verify this explicitly for your deployment and configure log retention accordingly.
HIPAA
HIPAA applies to covered entities and their business associates when Protected Health Information (PHI) is involved. PHI includes a broad set of identifiers (names, dates, geographic data, medical record numbers) when associated with health information. If your agents process clinical notes, patient records, or any data that could identify an individual in a health context, the entire inference stack is in scope.
Any vendor whose infrastructure could access PHI requires a signed Business Associate Agreement (BAA). The minimum necessary standard means agents should be designed to request and receive only the PHI required for the specific task. For HIPAA workloads, the strongest architecture is fully self-hosted inference where PHI never transits a third-party network; the question of whether a cloud GPU provider qualifies as a business associate depends on the specifics of their data access and your BAA terms.
EU AI Act
The EU AI Act (Regulation 2024/1689) entered into force in August 2024, with obligations phasing in through 2026. AI agents fall into scope depending on their application area. Systems used in healthcare, employment decisions, credit assessment, law enforcement, or critical infrastructure are classified as high-risk and carry significant requirements: documented risk management processes, data governance controls, transparency obligations, human oversight mechanisms, and accuracy and robustness standards.
General-purpose AI models used as the backbone of agent systems have their own transparency requirements under the Act. If you're building agents for any of the listed high-risk categories, compliance planning needs to happen at the system design stage, not after deployment.
Certifications: what they cover and what they don't
SOC 2 Type II, ISO 27001, and PCI DSS are the most common certifications you'll see from cloud infrastructure providers. SOC 2 Type II covers security, availability, processing integrity, confidentiality, and privacy controls over a defined audit period; it's the standard baseline for cloud infrastructure. ISO 27001 certifies an information security management system. PCI DSS is specific to payment card data and only relevant if agents process cardholder information.
Holding these certifications does not itself satisfy GDPR or HIPAA compliance; they demonstrate security controls are in place, which is a prerequisite, not a substitute, for regulatory compliance. Always confirm which certifications apply to the specific infrastructure tier your workload runs on, not the provider's organisation as a whole.
Runpod Secure Cloud
Runpod Secure Cloud infrastructure partners maintain SOC 2, ISO 27001, and PCI DSS certifications. For GDPR, HIPAA, and other regulatory requirements, contact Runpod's sales team directly to confirm current compliance coverage and available agreements for your specific use case; requirements and offerings vary and should be verified before making architecture decisions based on them.
On-premises
For workloads with the strictest data residency requirements — where data must physically stay within a specific jurisdiction and cannot transit through a cloud provider's infrastructure — on-premises GPU hardware remains the only fully private option. For most teams with privacy requirements, Runpod Secure Cloud provides sufficient isolation at a fraction of the cost and operational complexity of on-premises deployment
Frequently asked questions
What's the minimum VRAM needed to run a private AI agent?
The minimum practical setup is a GPU with 8GB VRAM running a small quantized model — a 7B model at 4-bit quantization fits in approximately 4–5GB, leaving headroom for the KV cache. On Runpod, the RTX 3090 (24GB) or RTX 4090 (24GB) are the most common choices for self-hosted agent deployments: enough VRAM for 7B–13B models at FP16 or up to ~34B models with 4-bit quantization, at a cost that makes sense for production. For 70B models you’ll need a higher-VRAM GPU such as an A100 80GB — a 70B model at 4-bit still needs approximately 35–40GB of VRAM. If you only need a simple agent for document Q&A or straightforward reasoning tasks, a quantized 7B model on a smaller GPU tier is often sufficient and significantly cheaper than running a 70B model by default.
Can I run multiple agent sessions simultaneously on a single self-hosted GPU?
Yes — vLLM handles concurrent requests via continuous batching, serving multiple inference requests on a single GPU without requiring separate model instances. The limit is VRAM: the model weights occupy a fixed portion, and the remaining capacity is shared across concurrent request KV caches. On an RTX 4090 (24GB) running a quantized 8B model (~5GB weights), you have roughly 17–18GB remaining for concurrent request processing — enough to handle dozens of short parallel agent calls before throughput degrades. Monitor GPU memory utilization in the Runpod dashboard; if it's consistently above 90%, either reduce concurrency or move to a larger GPU tier.
When is Ollama appropriate and when should I use vLLM?
Ollama is a solid choice for local development, personal projects, and single-user setups where you're not handling concurrent requests. It's easier to install, runs on consumer hardware without Docker expertise, and is fine for testing agent behavior before moving to production. vLLM is the right choice for production: it handles concurrent requests properly through continuous batching (Ollama supports parallel requests but is significantly less efficient than vLLM’s continuous batching for production workloads), exposes a fully OpenAI-compatible API that every agent framework expects, and gives you precise control over GPU memory utilization and model serving parameters. If agents are serving real users or running automated workflows, use vLLM.
What happens to my self-hosted agent if the Runpod pod restarts or crashes?
If you're running on a persistent Pod without checkpointing, in-progress agent sessions are lost and need to restart from scratch. Three things protect against this: first, LangGraph's PostgreSQL checkpointer saves agent state to an external database at each graph node — if the pod restarts, the agent resumes from the last checkpoint rather than the beginning. Second, attach a Runpod network volume for any data the agent generates or caches; network volumes persist independently of pod lifecycle. Third, use Runpod Secure Cloud rather than community cloud for production workloads — Secure Cloud runs on Runpod-owned hardware with enterprise-grade uptime, reducing unplanned restarts. For stateless agents where each session is self-contained, pod restarts only affect in-flight requests — add retry logic in your orchestration layer to handle these cleanly.
Related Guides: Building and Deploying Multi-Agent AI on Runpod
Each topic in this guide has its own deep-dive article:
