Business Context
BrightMail, an email marketing platform, tested a new subject-line generator to improve click-through rate (CTR). The product manager wants a result that communicates not just whether the experiment “won,” but also the uncertainty around the estimated lift.
Problem Statement
Use a confidence interval for the difference in two proportions to quantify uncertainty in the experiment result and explain how you would communicate that uncertainty to stakeholders.
Given Data
| Group | Emails Delivered | Clicks | CTR |
|---|
| Control (current subject lines) | 18,400 | 1,472 | 8.00% |
| Treatment (new generator) | 18,100 | 1,611 | 8.90% |
Additional test settings:
| Parameter | Value |
|---|
| Confidence level | 95% |
| Significance level | 0.05 |
Requirements
- Compute the observed CTR in each group and the absolute lift.
- Calculate the standard error for the difference in proportions.
- Construct the 95% confidence interval for the treatment-minus-control CTR difference.
- State whether the interval includes 0 and what that implies statistically.
- Explain how you would communicate the result to a non-technical product stakeholder.
- Briefly comment on whether the result appears practically meaningful.
Assumptions
- Users were randomly assigned to control and treatment.
- Each delivered email is an independent trial.
- The normal approximation is appropriate because both groups have large sample sizes and enough clicks/non-clicks.
- No major deliverability issues or audience-mix shifts occurred during the test.
