Business Context
StreamCart, a subscription video platform, changed the homepage call-to-action button and wants to know whether the observed change in signup conversion is real or just random variation.
Problem Statement
You are given results from a 10-day A/B test comparing the old homepage (control) with the new homepage (treatment). Determine whether the change in signup conversion rate is statistically significant at the 5% level.
Given Data
| Group | Sample Size | Signups | Conversion Rate |
|---|
| Control | 18,420 | 2,211 | 12.00% |
| Treatment | 17,980 | 2,284 | 12.70% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates for both groups.
- Calculate the pooled proportion and the standard error for a two-proportion z-test.
- Compute the z-statistic and two-tailed p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Decide whether the observed lift is statistically significant or likely noise.
- Briefly explain what the result means for the rollout decision.
Assumptions
- Users were randomly assigned to control and treatment.
- Each user appears only once in the experiment.
- The normal approximation is valid because both groups have large sample sizes and enough conversions/non-conversions.
- No major instrumentation issues or sample-ratio mismatch occurred during the test.
