Problem
Scenario
You work on a work-management product and ran an A/B test on a growth change for two weeks. The result is not statistically significant, but the PM argues the directional lift is enough and wants to launch anyway. You need to advise whether the evidence supports shipping.
Question
How would you respond to the PM, and what framework would you use to decide whether to ship, extend, or stop the experiment? Explain how statistical power, minimum detectable effect, and guardrail metrics affect your recommendation.
What This Tests
- Interpreting non-significant A/B test results
- Using power and MDE to distinguish inconclusive from negative results
- Applying guardrails to launch decisions
- Avoiding peeking and post-hoc decision-making
Practicing as: Product Growth Analyst interview at DiscordHi, I'll play your Discord interviewer for the Product Growth Analyst role. Candidates describe these interviews as mixed and moderately difficult, so expect me to be professional and fair. Take your time with the question above and answer like we're in the room.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
