CartBridge is a large e-commerce marketplace (~18M monthly active users) optimizing its mobile checkout. A redesign reduces the number of form fields and introduces address auto-complete. The PM is excited because the A/B test shows a higher purchase conversion rate, but Finance and Risk want to know how big the lift could realistically be before approving a full rollout (the change also increases third‑party address-lookup costs).
You’re asked to explain and quantify the significance of confidence intervals: not just whether the effect is “statistically significant,” but what range of true effects is consistent with the data.
Using the experiment results below, compute and interpret a 95% confidence interval (CI) for the difference in conversion rates (treatment − control). Then connect that CI to a hypothesis test decision and to a business rollout decision.
| Metric | Control (A) | Treatment (B) |
|---|---|---|
| Users exposed | 84,732 | 85,104 |
| Purchases (conversions) | 10,252 | 10,684 |
| Observed conversion rate | 12.099% | 12.555% |
| Confidence level | $1-\alpha = 0.95$ | $1-\alpha = 0.95$ |
Notes: The randomization unit is a user. Each user is counted once (first exposure). Assume independence between users and that the normal approximation for proportions is acceptable.