Business Context
StreamHub, a video subscription platform, tested a new landing page intended to increase free-trial signup rate. The experiment ran for 10 days with user-level randomization.
Problem Statement
You are reviewing the experiment results and need to explain how confidence intervals should guide the decision, not just whether the p-value is below 0.05. Compute the confidence interval for the treatment effect and interpret what it says about uncertainty, statistical significance, and business impact.
Given Data
| Group | Sample Size | Signups | Signup Rate |
|---|
| Control | 18,500 | 2,035 | 11.00% |
| Treatment | 18,900 | 2,211 | 11.70% |
Additional assumptions:
| Parameter | Value |
|---|
| Confidence level | 95% |
| Significance level | 0.05 |
| Test type | Two-sided |
Requirements
- State the null and alternative hypotheses for the difference in signup rates.
- Compute the observed difference in proportions between treatment and control.
- Calculate the standard error for the confidence interval of the difference in proportions.
- Construct the 95% confidence interval for the treatment effect.
- Explain whether the result is statistically significant using the confidence interval.
- Interpret the interval in business terms: what range of true lift is plausible, and would you recommend rollout?
Assumptions
- Users were randomly assigned and each user appears once.
- The normal approximation is appropriate because both groups have large sample sizes.
- No major sample-ratio mismatch or instrumentation issues were detected.
- The primary metric is signup conversion rate, measured consistently across both groups.
