Business Context
StreamCart, a subscription video platform, changed its landing page headline and observed a higher signup rate over one week. The product manager wants to know whether the increase is real or just random variation.
Problem Statement
Use the experiment results below to determine whether the observed change in signup conversion is statistically meaningful or likely noise.
Given Data
| Group | Visitors | Signups | Conversion Rate |
|---|
| Control (old headline) | 18,450 | 2,214 | 12.00% |
| Treatment (new headline) | 18,120 | 2,302 | 12.70% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
| Confidence level | 95% |
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates and the observed lift.
- Calculate the pooled proportion and the standard error for a two-proportion z-test.
- Compute the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Decide whether the change is statistically significant at the 5% level.
- Explain whether the result is large enough to matter for the business, not just statistically significant.
Assumptions
- Users were randomly assigned to control and treatment.
- Each visitor is counted once, and outcomes are independent across users.
- The normal approximation is appropriate because both groups have large sample sizes and enough conversions/non-conversions.
- No major traffic-quality shifts or instrumentation issues occurred during the test.
