Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Diagnose Sample Ratio Mismatch

HardA/B Testing & Experimentation00:00
I
Practice interviewer
Your interviewer
In session
I
Interviewer

Welcome to your interview.

The question is on your right: Diagnose Sample Ratio Mismatch. Take a moment with it first.

Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes). Discussion and graded submissions share your five interviewer interactions, so spend them well.

You need to log in / sign up to chat or submit.

Experiment Readout With Allocation Imbalance

Scenario

You work on a digital product team and an A/B test has just come back with an unexpected imbalance between control and treatment traffic. The primary metric looks promising, but the allocation does not match what was planned, and there are signs of possible data quality issues in the readout.

Question

How would you handle sample ratio mismatch or other data quality issues in the experiment readout, and how would you decide whether the result is trustworthy enough to act on?