Business Context
BrightMail tested a new subject-line generator to improve email click-through rate (CTR). The marketing team wants to know whether the observed lift is statistically significant and, just as importantly, what that conclusion does and does not imply.
Problem Statement
You are given results from a randomized A/B test comparing the current subject line strategy (control) with a new generator (treatment). Determine whether the difference in CTR is statistically significant at the 5% level, then explain what statistical significance tells you and what it does not tell you.
Given Data
| Group | Emails Delivered | Clicks | CTR |
|---|
| Control | 50,000 | 2,400 | 4.80% |
| Treatment | 50,000 | 2,650 | 5.30% |
Additional information:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
Requirements
- State the null and alternative hypotheses.
- Compute the pooled click-through rate.
- Calculate the standard error and z-statistic for a two-proportion z-test.
- Compute the two-sided p-value and decide whether the result is statistically significant.
- Construct a 95% confidence interval for the CTR difference.
- Explain, in plain language, what statistical significance tells BrightMail.
- Explain what statistical significance does not tell BrightMail, including practical significance and causality caveats.
Assumptions
- Users were randomly assigned to control and treatment.
- Each delivered email is an independent trial.
- No major deliverability issues or tracking bugs affected one group more than the other.
- The normal approximation is appropriate because both groups have large sample sizes and many clicks/non-clicks.
