Private AI solutions for companies that can’t send data to the cloud: an on‑prem and air‑gapped playbook

Private AI solutions for companies that can’t send data to the cloud

In 2025, private cloud AI platforms represented about 23.1% of AI platform revenue, driven by data residency and air-gap needs, while hybrid grew fastest at a projected 24.3% CAGR through 2034 (Dataintelo market report). CTOs live that split every day. Product teams want GenAI features. Risk teams say the data can’t leave.

Here’s the core point: private AI is an architecture and an operating model, not a vendor SKU. You can ship real features in constrained environments, but only if you treat the whole thing like a regulated production system from day one.

Private AI solutions for companies that can’t send data to the cloud: what “private” really means

Most CTOs I talk to get stuck in a fake binary: cloud or on‑prem. Private AI has at least four deployment shapes, and each one changes security posture, cost, and delivery speed.

Here’s the definition I use with boards and auditors:

Private AI is any AI system where your organization controls data residency, runtime dependencies, and access paths end to end.

That definition is picky on purpose. “Private” systems still leak through telemetry, model downloads, package registries, or a vendor control plane if you don’t design for it.

Common shapes, from least to most restrictive:

• Private VPC in public cloud: Data stays in your account, but the system still depends on public endpoints.
• BYOC managed platform: Vendor runs a control plane, your data plane stays in your account.
• Air-gapped with diode: One-way export for logs and health metrics, no inbound internet.
• Fully air-gapped: No network connectivity of any kind.

TrueFoundry puts it bluntly: air-gapped is not a configuration flag. Air-gapped is an architecture where every runtime dependency already sits inside the enclave (TrueFoundry air-gapped AI). That framing stops the “just block egress” debate and forces the right design conversation.

A private AI stack usually includes:

• Compute: GPU nodes, schedulers, and capacity controls.
• Model layer: Base models, fine-tunes, quantized variants, and evaluation suites.
• Data layer: Document stores, vector indexes, feature stores, and lineage.
• App layer: RAG services, agent runtimes, and API gateways.
• Security layer: Identity, secrets, policy, and audit logs.
• Ops layer: Monitoring, incident response, and change control.

Private AI works when the stack has owners, SLOs, and a roadmap. Treat it like a product, not a science project.

How to choose between on‑prem, private cloud, and air‑gapped AI

CTOs need a decision tool that holds up in a steering committee, not a research lab. I use a simple model called the ZANUS Placement Test. The name is a reminder that “no cloud” is a constraint, not a strategy.

The ZANUS Placement Test (a decision matrix you can reuse)

Score each dimension 0 to 3. Add the score. Higher scores push you toward air-gapped.

Interpretation:

• 0 to 5: Private VPC or BYOC works.
• 6 to 10: Private cloud or on-prem with controlled egress.
• 11 to 15: Air-gapped or air-gapped with diode.

A lot of enterprises land in the middle. Dataintelo’s numbers match that reality. Public cloud still leads at 58.4% in 2025, but private and hybrid exist because constraints are real (Dataintelo market report).

A concrete scenario

A regional bank wants a call center copilot that reads customer notes and suggests next steps. The bank has strict data localization rules and a hard stance on customer PII leaving the network.

• Z score: 2
• A score: 2
• N score: 2
• U score: 1
• S score: 2

Total: 9. That bank should run inference on‑prem or in a private cloud, and keep training and evaluation inside the same boundary.

Persistence Market Research cites a real example in BFSI: Temenos and NVIDIA deployed an on-prem generative AI platform for banks in November 2024, aiming for real-time insights with regulatory compliance and explainable AI (Persistence Market Research). That’s the pattern to copy.

Air‑gapped private AI architecture: what changes, what breaks, and what to build first

Air-gapped work feels slow at first. Teams lose package installs, hosted observability, and managed identity. The trick is to treat the platform like a sealed factory line. Everything you need is inside, and change control is part of the design.

Model strategy for disconnected environments

Disconnected deployments work best with a portfolio, not one giant model. VDF.ai describes a common pattern: a general model in the 30B to 70B range, plus smaller specialist models, plus lightweight embedding models for retrieval (VDF.ai air-gapped deployments).

Quantization matters more than people expect. 4-bit and 8-bit formats like GPTQ, AWQ, and GGUF can cut hardware needs with acceptable quality loss for many enterprise tasks (VDF.ai air-gapped deployments).

A practical portfolio for a regulated enterprise enclave:

• General LLM: 30B to 70B for reasoning and drafting.
• Classifier model: Small model for routing, PII detection, and intent.
• Extractor model: Fine-tuned NER for entities like account numbers.
• Embedding model: For RAG retrieval and dedupe.

That portfolio also keeps costs sane. A 70B model shouldn’t be writing every email subject line.

Security and monitoring in sovereign deployments

Air-gapped doesn’t remove the need for monitoring. Air-gapped raises the bar.

Michael Hannecke notes that local semantic checks can reach 70% to 80% of cloud detection rates with models like Llama and Mistral, and that the bigger gap is enterprise integration with SIEM and SOC workflows (Sovereign AI agent security). That matches what I see. Security teams don’t fail on model choice. Security teams fail on missing runbooks.

Build these controls early:

• Prompt injection detection: Local checks, plus allowlists for tools and actions.
• RAG boundary controls: Document-level ACLs, and query-time filtering.
• Audit trails: Store prompts, retrieved chunks, tool calls, and outputs.
• SOC integration: New event schemas for AI actions and policy violations.

Dynamiq describes air-gapped LLM deployments in finance and healthcare, with the core promise of isolated environments that remove unauthorized access paths (Dynamiq air-gapped LLM security). Isolation shifts work onto your team. Plan for it.

Performance and cost: measure the right things

Private AI fails when teams buy GPUs before they define the user experience.

Inference has two regimes, prefill and decode. Benchmarks should track time to first token and tokens per output token, plus p95 and p99 under load.

Kiran Palla’s benchmarking guide calls out TTFT and TPOT as the metrics that map to user experience, and it pushes unit economics like dollars per trained token and wasted runs from instability (GPU benchmarking guide). That’s the mindset CTOs need.

A simple starting SLO set for an internal copilot:

• p95 TTFT: under 1.5 seconds
• p95 end-to-end latency: under 8 seconds for 300 tokens
• Availability: 99.5% monthly
• Quality gate: less than 2% policy violations in sampled outputs

Cost is often the forcing function. Presidio cites expert predictions that large enterprises can see public cloud AI costs reach $1 million per month, and that more companies will shift on-prem in 2025 to control spend (Presidio private AI). That number gets real fast with high-volume inference and long contexts. Treat cost like a first-class metric, not a finance afterthought.

If you want a practical way to track the trade-offs, use our Cloud Cost Estimator for baseline scenarios and then compare against on-prem depreciation and support costs (/tools/cloud-cost-estimator).

What private AI changes for engineering leadership, governance, and delivery

Private AI is a people problem disguised as a hardware problem.

Team design that works past the pilot

A private AI program needs clear ownership. I like a three-team split:

• AI Platform team (6 to 10 engineers): GPUs, clusters, model registry, serving, and observability.
• AI Product enablement team (4 to 8 engineers): RAG patterns, SDKs, templates, and evaluations.
• Domain squads (2 to 6 squads): Actual features, workflows, and adoption.

Platform teams lose when they behave like ticket queues. Platform teams win when they ship paved roads, with a roadmap and an intake process.

If your org already runs platform engineering, connect this work to our guide on platform team charters and service ownership (internal link: /platform-team-charter-and-service-ownership).

Governance that doesn’t block shipping

SUSE cites a Forrester prediction that in 2025, 40% of regulated enterprises will unify data and AI governance (SUSE private AI trends). Unification matters because fragmented governance creates shadow deployments.

A workable governance model has three layers:

• Data governance: Classification, retention, and access controls.
• Model governance: Approved models, evaluation results, and change control.
• App governance: Allowed tools, allowed actions, and audit requirements.

Tie governance to delivery with gates engineers can actually pass:

• A model ships after it passes a fixed eval suite.
• A copilot ships after it passes red-team prompts and policy checks.
• A workflow ships after it has an owner and an incident runbook.

Use our Incident Postmortem template to build the muscle early (/tools/incident-postmortem). Private AI incidents include data exposure, but also silent quality regressions.

Procurement and vendor strategy

Private AI stacks often mix open models, commercial support, and enterprise platforms. CTOs need a clean build vs buy story, or procurement will run the program for you.

Use our Build vs Buy Matrix to structure the decision (/tools/build-vs-buy-matrix). The matrix helps you separate:

• Model hosting and serving
• Vector search and document stores
• Guardrails and evaluation tooling
• Observability and audit

A common pattern:

• Buy the GPU stack and support contracts.
• Build the domain RAG layer and workflows.
• Buy or adopt open tooling for evals, then standardize.

Track the whole portfolio in Command Center, so risk and cost stay visible (/command-center).

And don’t skip architecture docs. Air-gapped systems need clear diagrams for auditors and operators. Use our ArchiMate Modeler to keep views consistent across teams (/tools/archimate).

CTO recommendations: what to do in the next 30, 90, and 180 days

Immediate Actions (next 30 days)

1. Classify data and use cases: Map top 10 GenAI use cases to data classes, like PII, PCI, PHI, export-controlled.
2. Run the ZANUS Placement Test: Score each use case and pick a deployment shape.
3. Define SLOs and cost targets: Set TTFT, p95 latency, availability, and a dollars-per-1,000-requests target.
4. Pick a model portfolio: Select one general model, one embedding model, and one small classifier.

Policy Framework (next 90 days)

1. Model approval process: Require eval results, license review, and a rollback plan.
2. Prompt and data retention rules: Decide what you log, how long, and who can access it.
3. SOC runbooks for AI events: Add alert types for prompt injection, policy violations, and tool misuse.

Architecture Principles (next 180 days)

1. Dependency sealing: Mirror package registries, container images, and model artifacts inside the boundary.
2. Least-privilege tool calling: Limit agent actions to explicit tools with scoped permissions.
3. Evaluation as a release gate: Treat eval suites like unit tests, and run them on every model change.
4. Hybrid burst only with clean boundaries: Keep sensitive data inside, and burst with synthetic or anonymized workloads.

Every exec review hits the same question: can an on‑prem or air‑gapped setup match cloud model quality? Strong results are realistic for a lot of enterprise tasks, but scope control, retrieval quality, and evaluation discipline matter more than people want to admit.

Bigger picture: private AI is becoming normal infrastructure

Market data points in the same direction. Cloud-based AI platforms still dominate at about 67.4% share in 2025, but on-prem and hybrid keep growing as enterprises balance sovereignty and scale (Persistence Market Research). Dataintelo’s forecast shows hybrid growing fastest, which matches the reality of mixed constraints across business lines (Dataintelo market report).

Private AI also changes the talent profile inside your company. You need engineers who can run production systems, not only notebooks. Security leadership has to understand prompt injection and tool abuse, not only network segmentation.

So here’s the question I’d bring to your next staff meeting: which two workloads in your company deserve a sealed boundary, and which two should stay in the public cloud for speed and cost?

Sources

1. Presidio: On-Premise vs. Public AI: Why Businesses Are Choosing Private AI Solutions
2. Dataintelo: Cloud AI Platform Market Research Report 2034
3. SUSE Communities: Future Trends in Private AI: What’s Next for Secure and Scalable AI
4. Persistence Market Research: Artificial Intelligence Platform Market Growth 2032
5. Dynamiq: Mastering LLM Security: An Air-gapped Solution for High Security Deployments
6. VDF.ai: Air-Gapped AI Deployments: Running Enterprise AI Agents in Disconnected and Restricted Networks
7. Medium: Air-Gapped AI Security: Sovereign Deployments
8. TrueFoundry: Air-Gapped AI: Deploying LLMs in Defense & Regulated Finance
9. LinkedIn: Benchmarking AI workloads on GPUs (Kiran Palla)