Retention Lift Experiment for New Feature

Business Context

StreamHub, a subscription video app, launched a new personalized reminder feature. Product leadership wants to know whether it improves true user retention rather than only creating a short-lived spike in opens or clicks.

Problem Statement

Design and analyze an experiment that tests whether the feature increases 28-day retention. Use 7-day engagement only as a diagnostic metric, not the decision metric.

Given Data

A 6-week randomized controlled experiment was run on newly eligible users.

Additional design inputs:

Requirements

1. Define the primary metric, null hypothesis, and alternative hypothesis.
2. Explain why 28-day retention should be the primary success metric instead of 7-day engagement.
3. Test whether the observed 28-day retention lift is statistically significant using a two-proportion z-test.
4. Compute a 95% confidence interval for the retention lift.
5. Estimate the required sample size per group for detecting a 1.5 percentage point lift at 80% power.
6. State whether StreamHub should roll out the feature now, and list at least three experimental design safeguards to ensure the effect is causal.

Assumptions

• User-level randomization was implemented correctly.
• Each user appears once and outcomes are independent across users.
• No major concurrent launches affected retention during the test window.
• Retention is defined as at least one active session during days 22-28 after assignment.