Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Common Pitfalls in Experiment Results

HardA/B Testing & Experimentation00:00
I
Practice interviewer
Your interviewer
In session
I
Interviewer

Welcome to your interview.

The question is on your right: Common Pitfalls in Experiment Results. Take a moment with it first.

Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes). Discussion and graded submissions share your five interviewer interactions, so spend them well.

You need to log in / sign up to chat or submit.

Scenario

You work on a digital product team that runs frequent A/B tests and often sees conflicting reads from the same experiment depending on when the results are checked. Some launches look strong at first, then flatten out, while others show wins in the primary metric but raise concerns in related metrics.

Question

What are the common pitfalls you watch for when interpreting experiment results, and how do they change your confidence in whether a test should ship? Focus on the issues that can make a result look better or worse than it really is.