Business Context
BrightMail, a marketing automation company, tested a new subject line to improve email click-through rate (CTR). The analytics team wants to know whether the observed lift is statistically significant or could be due to random variation.
Problem Statement
You are given results from a randomized A/B test comparing the current subject line (control) with a new subject line (treatment). Determine whether the treatment's higher CTR is statistically significant at the 5% level.
Given Data
| Group | Emails Delivered | Clicks | Click-Through Rate |
|---|
| Control | 18,500 | 1,554 | 8.40% |
| Treatment | 17,900 | 1,665 | 9.30% |
Use a two-proportion z-test with a two-sided alternative.
Requirements
- State the null and alternative hypotheses.
- Calculate the sample click-through rates for both groups.
- Compute the pooled proportion under the null hypothesis.
- Calculate the standard error for the difference in proportions.
- Compute the z-statistic and two-sided p-value.
- Decide whether the result is statistically significant at α=0.05.
- Briefly explain what the result means for the product marketing team.
Assumptions
- Users were randomly assigned to control and treatment.
- Each delivered email is an independent trial.
- The normal approximation is appropriate because both groups have large sample sizes and sufficient numbers of clicks and non-clicks.
- No major deliverability issue affected one group differently from the other.
