AB Analysis Case

Walk me through an A/B analysis project. How did you use data to prove the feature was worth building? Which metrics did you look at? If a particular metric missed expectations, what could explain it?

Asked in the Round 4 stage.

Answer as a practical experiment review. Explain how you would turn an ambiguous feature request into a causal hypothesis, select a primary metric and guardrails, determine the minimum detectable effect and required sample, and make a ship decision. State assumptions explicitly because no product context, traffic volume, baseline, or metric definitions are provided.

1. Define the null and alternative hypotheses, primary metric, MDE, and guardrails.
2. Choose the randomization unit and describe allocation, duration, and power analysis.
3. Pre-register the analysis plan, including statistical testing, peeking, and multiple comparisons.
4. Diagnose an underperforming metric and distinguish product, measurement, targeting, and experiment-design causes.
5. Give a ship, iterate, or do-not-ship rule.