Problem
Scenario
You work on a digital learning product and your team has launched a new feature that is intended to help students come back more often. Early usage looks promising, but retention is noisy and leadership wants to know whether the feature truly improved user retention rather than just creating a short-term engagement bump.
Question
How would you evaluate whether the new feature actually improved retention? Walk through the experiment you would run, how you would measure success, and how you would decide whether to ship.
What success means
Experiment risks
- Retention is lagging and noisy
- Early engagement may reflect novelty rather than durable value
- Assignment or logging bugs can create sample ratio mismatch
- Repeated checking can inflate false positives
Practicing as: Data Scientist interview at ScaleHi, I'll play your Scale interviewer for the Data Scientist role. Candidates describe these interviews as mixed and moderately difficult, so expect me to be professional and fair. Take your time with the question above and answer like we're in the room.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
