Business Context
StreamCart, a subscription video platform, tested a new signup page intended to increase trial-start conversion. Product leadership wants to know not just whether the treatment beat control, but how to interpret the confidence interval for the lift before deciding on a full rollout.
Problem Statement
You are given results from a 2-week randomized A/B test. Estimate the 95% confidence interval for the difference in conversion rates and explain what that interval means in the context of the product decision.
Given Data
| Group | Sample Size | Trial Starts | Conversion Rate |
|---|
| Control (old signup page) | 18,400 | 2,208 | 12.0% |
| Treatment (new signup page) | 18,100 | 2,353 | 13.0% |
The team will use a two-sided 95% confidence interval and a significance level of α=0.05.
Requirements
- Compute the sample conversion rate for each group and the observed lift.
- Calculate the standard error for the difference in proportions used in the confidence interval.
- Construct the 95% confidence interval for the treatment minus control conversion difference.
- Determine whether the interval implies statistical significance at the 5% level.
- Interpret the interval correctly in business terms.
- Explain one common misinterpretation of a confidence interval in an experiment.
Assumptions
- Users were randomly assigned to control and treatment.
- Each observation is independent at the user level.
- The normal approximation is appropriate because both groups have large sample sizes and sufficient successes/failures.
- No major instrumentation issues or sample-ratio mismatch occurred during the test.
