Analyzing A/B Test Data

HardA/B Testing & Experimentation00:00

Practice interviewer

In session

5 left

00:00

Your question is Analyzing A/B Test Data. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

You need to log in / sign up to chat or submit.

Problem

Analyze a product A/B test, given a set of data.

Use the supplied dataset and accompanying experiment context. Do not assume the test is valid without checking allocation, metric definitions, exposure quality, and the analysis unit.

State the hypothesis, primary metric, MDE, and guardrails.
Analyze the treatment effect with confidence intervals and an appropriate statistical test.
Check power, sample ratio mismatch, peeking, multiple comparisons, and unit-of-randomization issues.
Explain whether the result is statistically and practically significant.
Make a ship, iterate, or do-not-ship recommendation, including what additional analysis is needed.