Business Context
BrightCart, an online retailer, tested a new promotional email subject line to improve purchase conversion from a marketing campaign. Users were randomly assigned to receive either the current subject line (control) or the new subject line (treatment).
Problem Statement
Determine whether the new subject line produced a statistically significant improvement in conversion rate, and quantify the size of the lift.
Given Data
| Group | Sample Size | Purchases | Conversion Rate |
|---|
| Control | 18,500 | 1,295 | 7.00% |
| Treatment | 18,200 | 1,401 | 7.70% |
Use a two-sided test with significance level α=0.05.
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates and the observed lift.
- Calculate the pooled proportion and standard error for a two-proportion z-test.
- Compute the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Decide whether BrightCart should roll out the new subject line to all campaign traffic.
Assumptions
- Random assignment was implemented correctly.
- Each user appears once and outcomes are independent across users.
- The normal approximation is appropriate because both groups have sufficiently large sample sizes and enough conversions/non-conversions.
- No major campaign execution issues affected one group differently from the other.
