From Chat to Control: AI Agents Are Hitting the Telemetry and Experimentation Wall

AI adoption inside engineering orgs is entering a new phase. The next wave is not another model upgrade, it is AI systems that touch production workflows: investigating incidents, proposing mitigations, informing A/B decisions, and automating data work. CTOs now face a sharper question: what architecture makes AI safe and useful when it can influence operational outcomes?

Several sources point to the same constraint from different angles: the bottleneck is trustworthy inputs and measurable outputs. dbt’s write-up on why agentic projects fail argues that many deployments “crash and burn” because the data foundation is weak, not because the agent logic is missing (dbt Blog, “Why agentics projects fail and how to fix them”). Snowflake’s “zero-copy data architecture” narrative reinforces the push to reduce duplication and cost while expanding access to more data types, including unstructured data used by enterprise AI (Snowflake Blog, “Zero-Copy Data Architecture: Snowflake Summit 2026”). The pattern is clear, agents amplify whatever data hygiene already exists.

Operational AI adds another layer: observability becomes an API surface for machines, not only dashboards for humans. Snowflake’s Observe announcement explicitly frames telemetry access as something agents can consume directly via an MCP server and CLI to speed incident investigation (Snowflake Blog, “Connect AI Agents to Telemetry with Observe MCP & CLI”). InfoQ’s coverage of Cloudflare migrating cdnjs to its developer platform highlights how modern primitives (Workers, KV, Durable Objects, Queues) get composed into systems that run at massive scale (InfoQ, “Cloudflare Migrates JavaScript CDN Serving 9B Requests a Day to Its Developer Platform”). That kind of composable platform architecture is exactly what AI-driven operations will lean on, because agents need predictable interfaces, bounded side effects, and high-fidelity signals.

Experimentation and decision-making are getting the same treatment, and Spotify’s A/B testing piece adds an important caution. LLM predictions can sometimes substitute for human outcomes, but only under explicit assumptions, and the validity risk is structural (Spotify Engineering, “When Can LLMs Replace Humans in A/B Tests?”). CTOs should read that as a governance requirement: when AI output becomes a proxy metric, the organization must document assumptions, monitor drift, and preserve a path to ground truth.

Three CTO takeaways follow.

First, treat “agent readiness” as a data and telemetry program. That program includes lineage, contracts, and incremental recomputation so agents do not reason over stale or inconsistent state (dbt’s emphasis on data foundations, plus dbt State’s incremental rebuild approach for cost and speed). Second, design an “operational API” for AI: curated telemetry schemas, rate limits, and permissions that allow investigation without allowing uncontrolled action (Snowflake Observe MCP direction). Third, formalize AI-in-the-loop evaluation: define when AI can recommend, when it can execute, and what rollback and audit look like, especially in experimentation where proxy outcomes can quietly change product direction (Spotify’s assumption warning).

Engineering leaders already know how to build reliable systems for humans. The emerging shift is building reliable systems for machine coworkers that read logs, query warehouses, and propose actions. The best architecture choices over the next year will be the ones that keep AI fast, bounded, and provable under pressure.

Sources

1. https://www.getdbt.com/blog/why-agentics-projects-fail-and-how-to-fix-them
2. https://engineering.atspotify.com/2026/8/when-can-llms-replace-humans-in-a-b-tests
3. https://www.snowflake.com/en/blog/observe-mcp-server-cli-ai-agents-telemetry/
4. https://www.snowflake.com/en/blog/snowflake-summit-2026-zero-copy-data-architecture/
5. https://www.infoq.com/news/2026/08/cloudflare-cdnjs-migration/
6. https://www.getdbt.com/blog/dbt-state-use-case