Business Context
BrightCart, an online retail marketplace, tested a new promotional email creative against the current version to improve purchase conversion from a campaign send.
Problem Statement
You need to determine whether the new email creative produced a statistically significant lift in conversion rate and whether the observed lift is large enough to justify rollout.
Given Data
| Group | Emails Delivered | Purchases | Conversion Rate |
|---|
| Control (current creative) | 24,800 | 1,364 | 5.50% |
| Treatment (new creative) | 25,200 | 1,537 | 6.10% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Confidence level | 95% |
Requirements
- State the null and alternative hypotheses for the A/B test.
- Compute the sample conversion rates and the observed lift in percentage points.
- Calculate the pooled proportion and the standard error for a two-proportion z-test.
- Compute the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Decide whether BrightCart should roll out the new email creative based on both statistical and practical significance.
Assumptions
- Users were randomly assigned to control and treatment.
- Each delivered email corresponds to one independent user opportunity.
- Conversion is binary: a user either purchased or did not purchase.
- The normal approximation is appropriate because both groups have large sample sizes and sufficient numbers of conversions and non-conversions.
