Business Context
StreamCart, a grocery delivery app, wants to test whether a new promotional email increases 7-day purchase conversion. The marketing team needs an experiment design and a quick read on whether the pilot result justifies a broader rollout.
Problem Statement
You are asked to design the experiment and evaluate the pilot A/B test result. Assume the primary metric is whether a user makes at least one purchase within 7 days of receiving the email.
Given Data
| Group | Users Assigned | Purchasers in 7 Days | Conversion Rate |
|---|
| Control (current email) | 8,000 | 640 | 8.0% |
| Treatment (new email) | 8,200 | 738 | 9.0% |
Additional design inputs:
| Parameter | Value |
|---|
| Baseline conversion rate | 8.0% |
| Minimum detectable effect | 1.0 percentage point |
| Significance level | 0.05 |
| Desired power | 0.80 |
| Test type | Two-sided |
Requirements
- Define the null and alternative hypotheses for the A/B test.
- Explain a sound experiment design: unit of randomization, primary metric, guardrail metrics, and key bias risks.
- Compute the observed treatment effect in percentage points.
- Run a two-proportion z-test using the pilot data.
- Construct a 95% confidence interval for the difference in conversion rates.
- Estimate the required sample size per group for detecting a 1.0 percentage point lift from an 8.0% baseline at 80% power.
- State whether the pilot result is statistically significant and whether you would recommend full rollout.
Assumptions
- Users were randomly assigned at the user level.
- Each user received at most one email during the test window.
- Outcomes are independent across users.
- No major deliverability issues or targeting changes occurred during the experiment.
