Problem
Scenario
You work on a digital product where an A/B test shows a clear improvement in the primary conversion metric for the treatment group. However, when you look at longer-term retention cohorts, the treatment appears worse than control. The team is unsure whether this is a real trade-off, noise from a lagging metric, or an analysis mistake.
Question
How would you interpret an experiment where the primary metric improved, but the long-term retention cohort got worse? How would you investigate whether the result is causal and decide whether to ship?
What this tests
- Balancing a primary metric against guardrail metrics
- Reasoning about lagging retention outcomes
- Power and MDE for a lower-frequency retention metric
- Diagnosing peeking, sample ratio mismatch, and instrumentation issues
Practicing as: Data Scientist interview at SlackHi, I'll play your Slack interviewer for the Data Scientist role. Candidates describe these interviews as mixed and moderately difficult, so expect me to be professional and fair. Take your time with the question above and answer like we're in the room.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
