AI’s Efficiency-First Phase: Routing Layers, Power Constraints, and the New Unit Economics

Investor sentiment and infrastructure reality are colliding with AI ambition. Market jitters around AI-linked chip makers suggest a tougher standard for ROI, and regulators are also starting to price the grid impact of new data centres. CTOs are walking into a period where “can we build it?” matters less than “can we operate it profitably, reliably, and within power limits?”

A cost and capacity squeeze is showing up in multiple places. The BBC reports concerns that AI euphoria is fading as chip stocks fall, a proxy for growing scrutiny on AI spend and payback timelines (BBC: financial reckoning). Another BBC piece describes potential up-front deposits of hundreds of millions tied to data-centre power demands, effectively turning electricity access into a gating factor for expansion (BBC: power deposits). Power is becoming a balance-sheet line item, not an ops footnote.

Engineering teams are responding by squeezing waste out of the stack and by making routing a core architectural concern. Snowflake’s case study on custom incrementalization claims a 99% reduction in Dynamic Table auto-clustering costs, a reminder that “default automation” can be financially dangerous at scale if workload patterns are not well matched (Snowflake: incrementalization). Snowflake also pushed Adaptive Compute GA across clouds, explicitly targeting variable and AI workloads with better price-performance without manual tuning (Snowflake: Adaptive Compute). Meanwhile, Microsoft published a three-layer LLM routing reference architecture for AI agents on AKS, formalizing model selection, traffic management, and execution paths as a distinct subsystem (InfoQ: routing architecture). A routing layer is quickly becoming the “load balancer era” moment for LLMs.

Data architecture is also being pulled toward low-latency efficiency rather than brute-force recomputation. Spotify describes indexing the data lake to support online point queries, blending lake economics with serving-time latency requirements (Spotify Engineering). AWS content on moving from batch to real-time recommendation systems reinforces the same direction: tighter feedback loops and serving architectures that reduce overprocessing and improve timeliness (AWS: batch to real-time). The practical takeaway is that “AI cost” is often “data movement and recomputation cost” wearing a different label.

CTO guidance for the next 6 to 12 months:

1. Treat model routing as a product surface. Build or adopt an explicit routing tier (policy, safety, cost controls, fallback models, caching) rather than embedding model choice in application code. Microsoft’s AKS reference architecture is a useful starting point for separating concerns and creating operational knobs.

2. Adopt AI unit economics that include power and latency. Track cost per successful task, cost per routed request class, and cost per incremental data refresh. Add power and capacity assumptions to planning, especially for regions where grid constraints can block growth.

3. Assume “automation defaults” are expensive defaults. Snowflake’s 99% cost reduction story is a case study in revisiting managed features when workload shape diverges from vendor assumptions. Put a quarterly review in place for the top three cost drivers in data and AI.

4. Re-architect data for selective freshness. Spotify’s indexing approach and AWS’s batch-to-real-time patterns both point toward a hybrid: keep the lake, but make serving paths explicit. Freshness should be tiered by business value, not evenly applied.

The teams that win in the next phase will not be the teams with the biggest model. The teams that win will be the teams that can route, serve, and refresh intelligently under cost and power constraints.

Sources

1. https://www.bbc.co.uk/news/articles/cevm09wkgz0o
2. https://www.bbc.co.uk/news/articles/c9q90q9qnn2o
3. https://www.infoq.com/news/2026/07/microsoft-agents-aks-routing/
4. https://www.snowflake.com/en/blog/thrive-dynamic-table-costs-custom-incremental/
5. https://www.snowflake.com/en/blog/adaptive-compute-ga-azure-google-cloud/
6. https://engineering.atspotify.com/2026/7/indexing-the-data-lake-for-online-point-queries
7. https://aws.amazon.com/blogs/big-data/building-a-scalable-personalized-recommendation-system-on-aws-from-batch-to-real-time/