Running AI on sensitive data without sending it to the public cloud requires deploying open-weight models within an environment you completely control, surrounding them with a robust data governance layer, and serving them on dedicated compute. A complete private AI architecture goes beyond merely downloading a model to a local machine; it demands a comprehensive stack where the data plane, model execution, identity management, and orchestration are fully integrated and strictly isolated from external networks.
(Disclosure: We make Hyperlake, the sovereign AI operating platform discussed in this article, which helps teams assemble, deploy, and govern these architectures in their own infrastructure or their clients' environments.)
Why bypass public cloud APIs for sensitive workloads?
Organizations dealing with proprietary codebases, regulated financial records, protected health information (PHI), or unreleased product designs often cannot transmit this data across network boundaries to multi-tenant, black-box vendor APIs.
While public cloud providers offer various compliance certifications and data privacy agreements, transferring data to an external API fundamentally shifts the control plane. Even when using confidential computing technologies (such as Intel TDX/SGX, AMD SEV-SNP, or NVIDIA Confidential Computing) to encrypt data in use, maintaining absolute data residency, data sovereignty, and verifiable network isolation usually requires architectural control over the entire stack.
When you run AI locally or in a private cloud environment, you retain full ownership of the model weights, the inference infrastructure, and the query logs. This eliminates the risk of external API logging, prevents your data from being inadvertently used in future vendor training runs, and allows you to enforce your organization's exact security perimeters.
What are the common tools for running models privately?
The ecosystem for running models outside the public cloud is divided into two primary categories: desktop-level applications for developers, and production-grade serving engines for enterprise workloads. Based on a historical survey of industry tools, here is how the landscape of local and private AI execution breaks down.
Comparison: Developer vs. Production Private AI Tools
While tools like Ollama and LM Studio are excellent for proving a concept on a laptop, they are not designed to handle concurrent multi-user enterprise traffic, strict identity-based data access controls, or distributed GPU orchestration. For that, you need a sovereign architecture.
What does a production-grade reference architecture look like?
A production-grade private AI deployment must safely connect large language models (like Llama, Mistral, or Qwen) to your enterprise data without exposing that data.
Note on this proposed architecture: The following is a hypothetical proposed reference architecture designed for sovereign control. Because no specific hardware benchmarks or independent latency studies are supplied in this article, actual token generation speed, accuracy rates, and concurrent user capacity will depend entirely on your specific GPU provisioning, network bandwidth, and the specific model sizes you choose to deploy. This method relies on standard open-source infrastructure patterns.
Layer 1: Infrastructure and Compute
The foundation of a portable, sovereign AI stack is Kubernetes. By abstracting the underlying hardware—whether it is bare metal on-premises, a private cloud, or an isolated AWS/Azure/GCP VPC—Kubernetes ensures that the services remain portable and manageable. To understand the specifics of hardware orchestration, see our guide on the sovereign Physical AI stack on Kubernetes.
Layer 2: Data and Knowledge Engines
AI is only as useful as the context it can access. Instead of copying data into isolated silos, a modern architecture relies on a governed data foundation. This typically involves:
- Analytics and Structured Data: Iceberg and Trino for lakehouse analytics, providing a central, queryable storage layer for large datasets. You can read more about how these components interact in our breakdown of the data lakehouse architecture.
- Transactional and Document Data: PostgreSQL for core application state and MongoDB for document storage.
- Vector and Graph Stores: Qdrant or Milvus for high-performance vector search (essential for Retrieval-Augmented Generation), and graph engines for mapping complex relationships.
- Event Streams: Kafka for processing real-time data feeds.
Layer 3: AI Services and Model Serving
Instead of calling a public API, your applications call internal endpoints. Open models are served through high-throughput engines like KServe or vLLM. This layer handles the loading of model weights into GPU memory, batching concurrent requests, and executing inference. This environment is also where fine-tuning and automated evaluation pipelines operate, grounded in your governed enterprise context.
Layer 4: Applications and Workflows
This is the layer where your AI-native products, enterprise agents, copilots, or domain-specific applications live. For advanced use cases like autonomous intelligence, this layer includes durable agent execution, state management, queues, and spending limits, ensuring an AI workforce operates with purpose, memory, and strict operational boundaries.
How do identity and data governance work in a sovereign stack?
The critical vulnerability in many AI deployments is the access control layer. If an AI agent has sweeping read access to an entire database, any user interacting with that agent can potentially extract data they are not authorized to see.
In a sovereign architecture, people and agents reach data through authenticated, governed services rather than shared database passwords. The flow operates as follows:
- Authentication: Users sign in through an OAuth/OIDC proxy connected to your existing enterprise identity provider.
- Identity Token: A JSON Web Token (JWT) is validated and passed along. This token carries scoped user or workload identity.
- Policy Evaluation: When the AI agent attempts to retrieve data (e.g., querying a vector database for documents to answer a prompt), the request hits an integrated access point. Here, Open Policy Agent (OPA) policy decisions are enforced.
- Enforcement and Audit: OPA checks the JWT against defined access policies. If the user does not have permission to view a specific document or data row, the semantic layer filters it out before it ever reaches the language model. Every access decision is logged for auditing and data lineage tracking.
As outlined in the NIST AI Risk Management Framework, managing AI risk requires mapping and measuring how data flows through a system. Hardcoding credentials or bypassing policy checks directly violates these risk management principles.
How do you operationalize and maintain this architecture?
Making models and applications work together across real data, security boundaries, compute, and client environments is the hardest part of AI in production. Repeatedly assembling this stack from scratch for every project leads to brittle infrastructure and massive integration overhead.
This is why we built Hyperlake. Hyperlake replaces repeated platform assembly with reusable deployment patterns, allowing teams to assemble, deploy, govern, observe, and maintain the entire lifecycle from a single control surface.
Instead of writing endless YAML files manually, infrastructure is callable through an application, a CLI, and MCP-compatible tools. A team states an intent in natural language—for example, "deploy a private model endpoint with governed access to asset data and a vector store in the customer's environment"—and Hyperlake proposes a plan. The team can inspect, approve, and deploy the changes, ensuring human approvals remain where policy requires them.
Hyperlake operates across AWS, Azure, Google Cloud, OVH, private clouds, and on-premises environments, allowing AI product companies to achieve repeatable installs and unified operations across many distinct client environments.
Operational resilience
Managing the lifecycle means handling backups, restores, patching, and scaling. Crucially, removing recovery protections or network boundaries in an attempt to save storage or compute overhead exposes the system to unrecoverable data loss and lateral network movement. Organizations must maintain proper backup protocols and network isolation, even if it requires additional operational overhead.
What about Physical AI and robotics?
Running AI on sensitive data is particularly critical in Physical AI—the intersection of software models and physical hardware like robotics or manufacturing systems. A Physical AI workload requires integrating simulation, orchestration, training data, and model services directly in the customer's cloud or on-premises environment.
A sovereign architecture enables this by integrating tools such as NVIDIA OSMO, Omniverse, Isaac Sim, and Isaac Lab (subject to licensing and validated integration) into a repeatable development environment. Because the data—often proprietary factory layouts or sensor feeds—never leaves the local infrastructure, intellectual property is strictly protected.
Does running private AI guarantee cost savings?
As agents multiply model calls, database searches, and background jobs, per-use pricing for public cloud APIs can rise exponentially with each action.
However, running private AI does not inherently guarantee cost savings. You are trading variable API costs for fixed infrastructure costs (GPU leasing or purchasing, power, and maintenance). Well-utilized owned capacity can make additional AI actions cost relatively little until more hardware capacity is needed, but idle compute is highly expensive.
Hyperlake helps mitigate this by ensuring eligible inference endpoints can scale down when idle, and by providing direct visibility into infrastructure usage and cost to right-size CPU and GPU capacity. Ultimately, savings in a sovereign architecture come from component reuse, reduced integration effort, and workload-aware capacity scaling, rather than a fixed percentage promise. The ROI case must be carefully evaluated and modeled per specific workload.
Frequently Asked Questions
What is sovereign AI?
Sovereign AI refers to artificial intelligence systems where the deploying organization retains absolute, exclusive control over the data, the model weights, the infrastructure, and the access policies. In a sovereign setup, no data or telemetry is sent to third-party API providers, ensuring strict data residency, privacy, and regulatory compliance.
Can I run models like Llama or Mistral completely offline?
Yes. Open-weight models like Meta's Llama series, Mistral, and Qwen can be downloaded and run completely offline. For individual developers, tools like LM Studio and Ollama facilitate this on desktop hardware. For enterprise production, these models are loaded into isolated Kubernetes clusters using serving engines like vLLM, completely disconnected from the public internet.
How do you control which data an AI agent can access?
Data access is controlled through an identity-to-data governance model. Instead of giving an AI model root access to a database, the system requires users to authenticate (via OAuth/OIDC). The user's identity token (JWT) is passed alongside their prompt. When the system retrieves contextual data from vector stores or databases to answer the prompt, an Open Policy Agent (OPA) enforces permissions, ensuring the AI only processes data the specific user is explicitly authorized to see.
Does running private AI guarantee cost savings over cloud APIs?
No, there is no guaranteed cost savings. While private AI prevents the runaway costs associated with per-token API billing (especially for highly active autonomous agents), it introduces the cost of provisioning, powering, and maintaining GPU infrastructure. Private AI is most economical when you have sustained, high-volume workloads that heavily utilize your owned capacity. If your infrastructure sits idle, private AI can be more expensive than paying for cloud APIs.
