Retention Lift A/B Test Design

Business Context

StreamHub is launching a new personalized re-engagement feature intended to improve 30-day user retention. The product team wants an experiment design that can both detect a meaningful lift and support a rollout decision.

Problem Statement

Design and analyze a randomized A/B test to measure whether the feature improves 30-day retention. Use the baseline retention and target lift below to size the experiment, then use the observed experiment results to test whether the feature worked.

Given Data

Assume a two-sided test and equal traffic allocation during planning.

Requirements

1. Define the null and alternative hypotheses for 30-day retention.
2. Compute the required sample size per group to detect the stated minimum lift.
3. Calculate the observed retention rates in control and treatment.
4. Run a two-proportion z-test using the observed data.
5. Compute a 95% confidence interval for the retention lift.
6. State whether the result is statistically significant and whether it is practically meaningful.
7. Briefly describe key experiment design choices: unit of randomization, primary metric, and at least two guardrails.

Assumptions

• Users are randomized at the user level and exposed consistently to one variant.
• Retention is defined as whether a user is active at least once during days 1-30 after assignment.
• No major concurrent launches materially affect retention during the test window.
• Independence between users is reasonable, and the normal approximation is valid due to large sample sizes.