The Art of CTO Experimentation Framework assesses experimentation maturity, helps design experiments with statistical rigor, and provides infrastructure gap analysis and sample size calculations for engineering teams.
Frequently Asked Questions
How do you build an experimentation culture in engineering?
Start with infrastructure: feature flags, experiment analysis tools, and guardrail metrics. Then change incentives: celebrate learnings from failed experiments, not just successful launches. Set expectations from leadership that a percentage of features should be validated through experiments. Create a low-friction process to propose and run experiments. The biggest blocker is usually not technical — it is cultural resistance to admitting uncertainty about what customers want.
What is the minimum sample size for a meaningful A/B test?
Sample size depends on three factors: baseline conversion rate, minimum detectable effect (MDE), and desired statistical significance. For a 5% baseline conversion rate with 2% MDE at 95% significance, you need approximately 50,000 users per variant. For a 50% baseline (like click-through rates) with 5% MDE, roughly 3,000 per variant. This is why most startups cannot run statistically valid A/B tests on low-traffic features — they should use qualitative methods or broader rollout strategies instead.