Feature Rollout Decision After Experiment

Business Context

StreamHub, a video platform, tested a new personalized home-feed ranking feature intended to increase 7-day retention. After a 21-day A/B test, the product team needs to decide whether to roll the feature out to all users.

Problem Statement

Use the experiment results to determine whether the feature produced a statistically significant improvement in 7-day retention and whether the effect is large enough to justify a full rollout.

Given Data

Additional rollout criteria:

Requirements

1. State the null and alternative hypotheses for a two-sided test.
2. Compute the sample retention rates and the observed lift.
3. Calculate the pooled proportion and standard error for a two-proportion z-test.
4. Compute the z-statistic and p-value.
5. Construct a 95% confidence interval for the retention-rate difference.
6. Decide whether StreamHub should roll out the feature to all users, using both statistical and practical significance.

Assumptions

• Users were randomly assigned and exposed to only one variant.
• Retention is measured once per user, so observations are independent.
• No major logging issues or sample-ratio mismatch occurred during the test.
• The test window is representative of normal user behavior.