A/B Testing Case Interpretation

A/B testing case study built around two plots: interpret the plots and reason about the experiment.

Asked in the Technical interview 1 (team) stage. Second half of the first team interview.

Using the two plots provided by the interviewer, explain what each plot shows and whether the experiment supports a change in the product.

State the null and alternative hypotheses, identifying whether the test is directional.
Define the primary metric, at least two guardrails, the unit of randomization, and the MDE you would use.
Interpret the difference between treatment and control, uncertainty intervals, time trends, and any imbalance or anomalous pattern in the plots.
Explain the required sample-size calculation, analysis plan, and checks for peeking, novelty, sample ratio mismatch, or a mismatch between randomization and analysis units.
Give a ship, do not ship, or iterate recommendation, including what additional evidence you would request.

A/B testing case study built around two plots: interpret the plots and reason about the experiment.

Asked in the Technical interview 1 (team) stage. Second half of the first team interview.

Using the two plots provided by the interviewer, explain what each plot shows and whether the experiment supports a change in the product.

State the null and alternative hypotheses, identifying whether the test is directional.
Define the primary metric, at least two guardrails, the unit of randomization, and the MDE you would use.
Interpret the difference between treatment and control, uncertainty intervals, time trends, and any imbalance or anomalous pattern in the plots.
Explain the required sample-size calculation, analysis plan, and checks for peeking, novelty, sample ratio mismatch, or a mismatch between randomization and analysis units.
Give a ship, do not ship, or iterate recommendation, including what additional evidence you would request.