Business Context
BrightCart ran an A/B test on a promotional email subject line. The marketing director does not want only a p-value; they want a confidence interval that can be explained clearly to non-technical stakeholders.
Problem Statement
Use the A/B test results below to estimate the lift in click-through rate (CTR) from the new subject line versus the old one, construct a 95% confidence interval for the difference in proportions, and explain how you would present that interval to executives.
Given Data
| Group | Emails Delivered | Clicks | Observed CTR |
|---|
| Control (old subject line) | 8,400 | 756 | 9.00% |
| Treatment (new subject line) | 8,100 | 810 | 10.00% |
Assume a two-sided significance level of 0.05.
Requirements
- State the parameter of interest and the null and alternative hypotheses.
- Compute the observed difference in CTR between treatment and control.
- Calculate the standard error for the difference in proportions.
- Construct the 95% confidence interval for the true CTR lift.
- Determine whether the result is statistically significant at the 5% level.
- Write a short stakeholder-friendly interpretation of the confidence interval.
- Briefly explain one common misinterpretation of a 95% confidence interval and the correct interpretation.
Assumptions
- Users were randomly assigned to subject lines.
- Each delivered email corresponds to one independent user outcome.
- Sample sizes are large enough for the normal approximation to the binomial distribution.
- No major deliverability issues differed across groups.
