Business Context
StreamCart, a subscription video platform, changed its signup page to reduce friction and ran a randomized A/B test. The product manager wants to know whether the observed change in signup conversion is real or just sampling noise.
Problem Statement
Determine whether the treatment page produced a statistically significant change in conversion rate versus the control page using a two-proportion z-test.
Given Data
| Group | Users Exposed | Conversions | Conversion Rate |
|---|
| Control (old signup page) | 18,500 | 2,220 | 12.00% |
| Treatment (new signup page) | 19,100 | 2,464 | 12.90% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
| Confidence level | 95% |
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates and the observed lift.
- Calculate the pooled proportion and pooled standard error.
- Compute the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Decide whether the change is statistically significant at the 5% level.
- Briefly explain whether the result is large enough to matter for the business.
Assumptions
- Users were randomly assigned to control and treatment.
- Each user appears once in the analysis.
- Conversion is binary and independent across users.
- Sample sizes are large enough for the normal approximation to be valid.
- No major instrumentation bugs or sample-ratio mismatch occurred during the test.
