Designing A/B Testing at Scale

Walk me through how you would design an experimentation platform or A/B testing framework for a high-traffic consumer service.

Address the end-to-end design, from experiment assignment and exposure logging through analysis and decision-making. Explain how your framework supports reliable metrics, power analysis, guardrails, monitoring, and safe operation at scale.

1. Define a representative hypothesis, primary metric, MDE, and guardrails.
2. Choose randomization and analysis units, then calculate sample size and duration using explicit assumptions.
3. Specify the statistical analysis plan, including peeking, multiple comparisons, SRM, and unit mismatch.
4. Describe how the platform detects interference, novelty effects, and other validity threats.
5. State a pre-registered ship, iterate, or do-not-ship rule.