Private AI solutions for companies that can’t send data to the cloud: a CTO playbook

Private AI solutions for companies that can’t send data to the cloud

Air-gapped and fully on-prem LLM deployments in large enterprises hit 14 to 19 percent in Q1 2026, up from about 7 percent in Q1 2025. Defence and intelligence lead spend at about 38 percent, then healthcare at 19 percent, regulated finance at 17 percent, and legal at 9 percent. Procurement cycles run 9 to 14 months for air-gapped programs, not weeks. Those numbers come from Presenc AI’s 2026 air-gapped deployment statistics.

CTOs get squeezed from both sides. Product teams want copilots, search, and agents yesterday. Security teams (often correctly) block cloud APIs because prompts and retrieval touch PII, PHI, source code, pricing, and trade secrets. Private AI can bridge the gap, but only if you build it like a platform you’ll run for years, not a demo that happens to work.

What are private AI solutions, and what “air-gapped” really means

“Private AI” gets used as a catch-all, and that’s where confusion starts. Auditors don’t like fuzzy terms, and neither do incident responders.

Here’s the definition I use with boards and security teams:

Private AI means you run model inference and data retrieval inside infrastructure you control, with enforceable boundaries on data egress, identity, and logging.

In practice, private AI usually shows up in a few shapes:

• On-prem inference. GPUs sit in your data center, your colo, or your cage.
• Private cloud inference. You run on a private cloud stack because ops, governance, and standardization matter. VMware’s 2026 report argues private cloud has moved to the center of production AI placement, driven by security, cost, performance, and business criticality. See Private Cloud Outlook 2026.
• BYOC in public cloud. You run inside your own cloud account, with strict controls. Many “no cloud” policies still allow this, but true air-gap rules don’t.
• Edge inference. You run close to devices for latency and data locality.

Air-gapped is a much stronger claim. Boyu Wang at TrueFoundry puts it plainly: air-gapped is an architecture where every runtime dependency already sits inside the enclave, with no route to the outside. See Air-Gapped AI: Deploying LLMs in Defense and Regulated Finance.

TrueFoundry also lays out a useful continuum of isolation tiers, from connected, to BYOC, to air-gapped with one-way export, to fully air-gapped. That tiering saves teams from building a bunker for a workflow that only needed a fence.

The practical move is to place workloads based on data sensitivity, latency, and cost. Several 2026 trend writeups land in the same place, including Stellium’s hybrid AI architecture notes and Cloudera’s 2026 predictions on hybrid convergence and control planes.

How to choose between on-prem, private cloud, BYOC, and air-gap

You need a decision tool that survives both security review and budget review. If the framework falls apart the moment someone asks “prove it,” it’s not a framework.

I use a simple model called the ZANUSAI Gate. The name sticks, and the checks map to real constraints.

ZANUSAI Gate: Zero egress, Audit, Need, Users, Spend, Availability, Integration

Pass the gate in order. Stop when a gate fails.

The Spend gate lines up with DataCouch’s “On-Premises Switch” guidance. DataCouch calls out three common triggers: security blockers, token spend above $10,000 per month, and latency targets like TTFT under 100ms. See On-Premises LLM Inference and Generative AI Infrastructure.

The question that shows up in every steering meeting is predictable: do we really need full air-gap? Most teams don’t. Most teams need a private boundary with tight egress controls and serious governance.

Reference architecture for private AI that works in regulated companies

Private AI programs fail for boring reasons. Teams treat the model as the product and ignore the platform work that keeps the thing running.

A private AI stack that holds up in regulated environments usually has five layers:

• Model runtime. vLLM, TGI, llama.cpp, or a vendor runtime. Keep the runtime behind an internal API.
• Retrieval layer. RAG pipelines, embeddings, vector store, and document permissions.
• Guardrails and evaluation. Prompt filtering, output policies, red teaming, and regression tests.
• Observability and cost. Token counts, latency, GPU utilization, and per-team chargeback.
• Governance and identity. SSO, RBAC, audit logs, and change control.

Hardware sizing and the “2 to 8 H100” reality

Presenc AI reports the most common air-gapped shape as an on-prem GPU server with 2 to 8 H100s, running open-weight base models like Llama, Qwen, and Mistral, with internal fine-tunes. See Presenc AI’s statistics.

That matches what I see. One 8x H100 box becomes the first shared inference node. Then the platform team adds a second box so patching and maintenance don’t turn into an outage.

Cost conversations get messy fast, especially once chargeback enters the chat. Lyceum’s Europe benchmark lists H100 virtual machines at $2.79 per GPU-hour on demand and $3.59 per GPU-hour for dedicated inference and serverless training. See Lyceum’s 2026 inference latency benchmark. Those numbers are handy for sanity-checking vendor quotes and internal pricing.

Latency targets that product teams will accept

Private AI can win on latency because you cut network hops and avoid cold starts. DataCouch calls out sub-100ms TTFT as a target for real-time apps. See DataCouch.

Product teams usually care about two metrics:

• Time to first token. Users feel this.
• Tokens per second. Finance feels this.

Set SLOs early, before the first “why is it slow today?” escalation. Our guide to SLOs and error budgets in Command Center can help you track those targets alongside incidents and capacity (/command-center).

Governance is not paperwork, it is the product

UnitedLayer makes the point bluntly: governance is what separates the teams that ship from the teams that spend a year cleaning up shadow deployments. See Why 2026 Marks AI’s Transition From Hype To Enterprise Reality.

Governance needs mechanisms you can point to during an audit and rely on during an incident:

• Model registry with approved versions and owners.
• Prompt and retrieval logging with retention rules.
• Access reviews tied to HR events.
• Change control for model updates and embedding refresh.

Our incident postmortem template works well for model regressions and data leaks because the failure modes behave like incidents, not “AI quirks” (/tools/incident-postmortem).

Privacy enhancing tech is real, but don’t bet the roadmap on it

AgileSoftLabs highlights privacy-enhancing technologies like homomorphic encryption, secure multi-party computation, and trusted execution environments. See Why Private AI is the Future of Secure and Ethical Enterprise AI Adoption.

PETs can help in narrow cases. PETs don’t replace basics like egress blocks, key management, and audit logs. Treat PETs as phase two, not the thing you need before you can ship anything.

Enterprise implications for CTOs: cost, risk, and org design

Private AI changes how you build systems and how you run teams. The tech is only half the story.

1. Shadow AI becomes a supply chain problem

Teams will route data to cloud tools if you block them without a workable alternative. UnitedLayer calls out shadow IT silos as audit nightmares. See UnitedLayer.

A private AI platform gives teams a sanctioned path. Security gets a single choke point, and engineering gets something they can actually build on.

2. Procurement timelines will break product roadmaps

Presenc AI reports 9 to 14 months from vendor evaluation to production for air-gapped deployments. See Presenc AI.

That timeline forces a two-track plan:

• A short-term track for low-risk use cases in a controlled environment.
• A long-term track for air-gapped or sovereign builds.

3. FinOps moves from cloud bills to GPU capacity planning

Cloud APIs hide the hardware. Private AI puts hardware right in front of you, along with all the awkward questions about utilization.

You need a cost model product and finance both trust. Start with a simple unit model:

• Cost per million tokens
• Cost per document indexed
• Cost per active user per month

Our cloud cost estimator can help you compare private cloud and public cloud baselines, even if the final answer ends up on-prem (/tools/cloud-cost-estimator).

4. Platform ownership becomes a leadership test

Private AI needs a team that owns uptime, upgrades, and guardrails. A part-time “AI guild” won’t hold the pager.

Treat the private AI stack like an internal product. Publish SLOs, keep a roadmap, run office hours, and say “no” to one-off deployments that bypass the platform. Our writing on platform teams as internal products fits this pattern (see our internal post on platform team operating models).

CTO recommendations: what to do in the next 90 days

Most companies start with model selection and end up stuck. Start with constraints and workflows, then pick the tech.

Immediate actions

1. Classify use cases by data class. Tag each use case as public, internal, confidential, regulated. Tie tags to deployment tiers.
2. Pick one “thin slice” workflow. Choose a workflow with 50 to 200 users and measurable time savings. Legal clause search and IT ticket triage work well.
3. Stand up a private RAG pilot. Keep data retrieval inside your network. Run inference on-prem or in private cloud.
4. Set two SLOs. Track TTFT and error rate. Put the SLOs in Command Center so leadership sees drift (/command-center).
5. Run a red team day. Attack prompt injection, data exfil via tool calls, and permission bypass. Write a postmortem for every finding (/tools/incident-postmortem).

Policy framework

1. Deployment tier policy. Map data classes to connected, BYOC, private cloud, air-gapped. Use TrueFoundry’s tier language so auditors understand the boundary. See TrueFoundry.
2. Model approval policy. Require an owner, a rollback plan, and an eval suite before promotion.
3. Logging and retention policy. Decide what you log, where you store it, and who can read it. Security will ask anyway.

If your org struggles with policy sprawl, use our ArchiMate Modeler guide to document the control points and data flows in a way auditors can follow (/tools/archimate).

Architecture principles

1. Boundary-first design. Put an internal gateway in front of all model calls. Block direct outbound calls from apps.
2. Least privilege retrieval. Enforce document permissions at query time, not only at index time.
3. Offline build pipeline for air-gap. Mirror container registries, Python wheels, model weights, and OS patches into the enclave.

A build vs buy decision shows up fast. Use our Build vs Buy Matrix to decide if you should run your own inference stack or buy a managed private platform (/tools/build-vs-buy-matrix).

Bigger picture: private AI is becoming the default for high-stakes work

VMware’s 2026 report frames private cloud as the center of production strategy for high-stakes workloads, driven by security, cost, performance, and geopolitical and regulatory scrutiny. See Private Cloud Outlook 2026.

Cloudera’s 2026 predictions point to hybrid convergence and a unified control plane, since AI agents need consistent access across environments. See Cloudera’s 2026 predictions.

Private AI also shifts the talent profile you need. You want engineers who can run GPUs, Kubernetes, storage, and security controls. Hiring one “prompt engineer” won’t change much. Build a small platform team, then train app teams to consume the platform safely.

A cloud AI ban without a private alternative leads to predictable behavior: teams route around the ban, and the audit trail turns into Swiss cheese. Which deployment tier does your highest-risk workflow belong in, and who owns it end to end?

Sources

1. Presenc AI, Air-Gapped LLM Deployment Statistics 2026
2. TrueFoundry, Air-Gapped AI: Deploying LLMs in Defense and Regulated Finance (May 12, 2026)
3. VMware, Private Cloud Outlook 2026: The AI Tipping Point
4. DataCouch, On-Premises LLM Inference and Generative AI Infrastructure
5. UnitedLayer, Why 2026 Marks AI’s Transition From Hype To Enterprise Reality
6. Cloudera, 2026 Predictions: Architecture, Governance, and AI Trends
7. Stellium Consulting, 2026 AI Trends: What Enterprises Need to Know
8. AgileSoftLabs, Why Private AI is the Future of Secure and Ethical Enterprise AI Adoption (2026)
9. Lyceum, 2026 LLM Inference Latency Benchmark: Europe GPU