Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Test Retention Lift from New Feature

HardA/B Testing & Experimentation00:00
Practice interviewer
In session
5 left
00:00

Your question is Test Retention Lift from New Feature. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

You need to log in / sign up to chat or submit.

Problem

Scenario

You work on a digital learning product and your team has launched a new feature that is intended to help students come back more often. Early usage looks promising, but retention is noisy and leadership wants to know whether the feature truly improved user retention rather than just creating a short-term engagement bump.

Question

How would you evaluate whether the new feature actually improved retention? Walk through the experiment you would run, how you would measure success, and how you would decide whether to ship.

What success means

MDE·2 percentage points absoluteUnit·user_idBaseline·40%Primary metric·28-day retention

Experiment risks

  • Retention is lagging and noisy
  • Early engagement may reflect novelty rather than durable value
  • Assignment or logging bugs can create sample ratio mismatch
  • Repeated checking can inflate false positives