Early Stopping in A/B Tests

EasyStatistics & Probability00:00

Practice interviewer

In session

5 left

00:00

Your question is Early Stopping in A/B Tests. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

You need to log in / sign up to chat or submit.

Problem

Scenario

You're reviewing an experiment on a product change, and after two days the treatment already looks significant. The PM wants to call the test early and ship.

Question

A PM wants to stop an experiment after two days because it already looks significant. What would you recommend?

What this tests

Understanding of p-values and statistical significance
Why peeking inflates false positive rates
Judgment on early stopping in A/B tests
When sequential testing is appropriate