Retention Lift Significance for Onboarding Test

Business Context

Streamly, a mobile subscription app, launched a new onboarding flow intended to improve early user retention. A product manager wants to know whether the observed lift in 7-day retention is statistically meaningful or just random variation.

Problem Statement

You are given results from a randomized A/B test comparing the old onboarding experience (control) to the new onboarding experience (treatment). Determine whether the retention lift is statistically significant at the 5% level, and quantify the uncertainty around the lift.

Given Data

Additional test settings:

Requirements

1. State the null and alternative hypotheses.
2. Compute the sample retention rates and the observed lift.
3. Calculate the pooled proportion and standard error for a two-proportion z-test.
4. Compute the z-statistic and two-sided p-value.
5. Construct a 95% confidence interval for the retention-rate difference.
6. Conclude whether the lift is likely real or noise, and explain the business implication.

Assumptions

• Users were randomly assigned and each user appears once.
• Retention is measured consistently across both groups.
• The normal approximation is valid because both groups have large sample sizes and sufficient retained/non-retained counts.
• No major instrumentation changes occurred during the experiment.