Whenever you're ready, walk me through how you'd approach this.
BrightCart, an online retail platform, tested a new promotional email subject line to improve purchase conversion. The analytics team wants to know how to interpret the result when the confidence interval for lift includes zero.
A 10-day A/B test compared the current subject line (control) against a new subject line (treatment). Your task is to evaluate whether the treatment changed conversion rate and explain what it means if the confidence interval for the treatment effect crosses zero.
| Group | Sample Size | Purchases | Conversion Rate |
|---|---|---|---|
| Control | 8,200 | 656 | 8.00% |
| Treatment | 8,050 | 684 | 8.50% |
Additional test settings:
| Metric | Value |
|---|---|
| Observed lift | 0.50 percentage points |
| Significance level | 0.05 |
| Confidence level | 95% |