Business Context
Streamly, a mobile subscription app, launched a new onboarding flow intended to improve early user retention. A product manager wants to know whether the observed lift in 7-day retention is statistically meaningful or just random variation.
Problem Statement
You are given results from a randomized A/B test comparing the old onboarding experience (control) to the new onboarding experience (treatment). Determine whether the retention lift is statistically significant at the 5% level, and quantify the uncertainty around the lift.
Given Data
| Group | Users | Retained at Day 7 | Retention Rate |
|---|
| Control | 18,400 | 7,176 | 39.0% |
| Treatment | 18,950 | 7,770 | 41.0% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
Requirements
- State the null and alternative hypotheses.
- Compute the sample retention rates and the observed lift.
- Calculate the pooled proportion and standard error for a two-proportion z-test.
- Compute the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the retention-rate difference.
- Conclude whether the lift is likely real or noise, and explain the business implication.
Assumptions
- Users were randomly assigned and each user appears once.
- Retention is measured consistently across both groups.
- The normal approximation is valid because both groups have large sample sizes and sufficient retained/non-retained counts.
- No major instrumentation changes occurred during the experiment.
