Business Context
StreamHub, a video platform, tested a new personalized home-feed ranking feature intended to increase 7-day retention. After a 21-day A/B test, the product team needs to decide whether to roll the feature out to all users.
Problem Statement
Use the experiment results to determine whether the feature produced a statistically significant improvement in 7-day retention and whether the effect is large enough to justify a full rollout.
Given Data
| Group | Users | Retained at Day 7 | Retention Rate |
|---|
| Control (old feed) | 52,400 | 19,388 | 37.0% |
| Treatment (new feed) | 52,100 | 20,319 | 39.0% |
Additional rollout criteria:
| Metric | Threshold |
|---|
| Significance level | 0.05 |
| Minimum practical lift | 1.0 percentage point |
Requirements
- State the null and alternative hypotheses for a two-sided test.
- Compute the sample retention rates and the observed lift.
- Calculate the pooled proportion and standard error for a two-proportion z-test.
- Compute the z-statistic and p-value.
- Construct a 95% confidence interval for the retention-rate difference.
- Decide whether StreamHub should roll out the feature to all users, using both statistical and practical significance.
Assumptions
- Users were randomly assigned and exposed to only one variant.
- Retention is measured once per user, so observations are independent.
- No major logging issues or sample-ratio mismatch occurred during the test.
- The test window is representative of normal user behavior.
