Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Directionally Useful but Inconclusive Test

HardA/B Testing & Experimentation00:00
I
Practice interviewer
Your interviewer
In session
I
Interviewer

Welcome to your interview.

The question is on your right: Directionally Useful but Inconclusive Test. Take a moment with it first.

Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes). Discussion and graded submissions share your five interviewer interactions, so spend them well.

You need to log in / sign up to chat or submit.

Problem

Scenario

You work on a mobile game and have run an A/B test on a new onboarding flow. The result looks directionally positive, but the evidence is not strong enough to be statistically conclusive, and stakeholders want to know whether to ship, iterate, or keep testing.

Question

How do you decide what to do when research findings are directionally useful but not statistically conclusive? What would you look at before making a ship decision?