Business Context
BrightCart, an online retailer, tested a new promotional email subject line to improve open rates. The marketing team wants to know whether the observed lift is statistically significant or could plausibly be due to random variation.
Problem Statement
You are given results from a randomized A/B test comparing the current subject line (control) to a new subject line (treatment). Determine whether the treatment meaningfully changed the email open rate, explain what statistical significance means in this setting, and describe how the p-value relates to the decision.
Given Data
| Group | Emails Sent | Opens | Open Rate |
|---|
| Control | 8,400 | 1,764 | 21.0% |
| Treatment | 8,100 | 1,863 | 23.0% |
Additional test setting:
| Parameter | Value |
|---|
| Significance level (α) | 0.05 |
| Test type | Two-sided |
Requirements
- State the null and alternative hypotheses.
- Compute the sample open rates and the observed lift.
- Calculate the pooled proportion and standard error for a two-proportion z-test.
- Compute the z-statistic and two-sided p-value.
- Decide whether the result is statistically significant at the 5% level.
- Explain, in plain language, what the p-value means and what it does not mean.
- Compute a 95% confidence interval for the difference in open rates.
- Give a business recommendation on whether BrightCart should adopt the new subject line.
Assumptions
- Users were randomly assigned to one subject line.
- Each email outcome is independent at the user level.
- Open tracking is measured consistently across both groups.
- The normal approximation for proportions is appropriate because both groups have large sample sizes.
