Business Context
BrightNote, a productivity app, tested a new onboarding flow intended to improve free-to-paid signup conversion. A product manager wants a plain-English explanation of what the p-value and confidence interval mean for the launch decision.
Problem Statement
You need to analyze the A/B test and then explain the statistical results in language a non-technical product partner can use. Focus on both statistical significance and the likely size of the effect.
Given Data
| Group | Sample Size | Paid Signups | Conversion Rate |
|---|
| Control (old onboarding) | 18,400 | 2,208 | 12.0% |
| Treatment (new onboarding) | 18,100 | 2,353 | 13.0% |
Additional parameters:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Confidence level | 95% |
| Test type | Two-sided |
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates and the observed lift.
- Calculate the two-proportion z-test statistic and p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Explain, in plain English, what the p-value means.
- Explain, in plain English, what the confidence interval means.
- Make a recommendation on whether BrightNote should launch the new onboarding flow.
Assumptions
- Users were randomly assigned to control and treatment.
- Each user appears once in the experiment.
- The normal approximation is appropriate because both groups have large sample sizes and enough conversions/non-conversions.
