CartJet is a large e-commerce marketplace (~8M weekly active users) optimizing its mobile checkout. A redesign reduces the number of form fields and adds Apple Pay earlier in the flow. Finance estimates that each additional completed order is worth $6.40 in contribution margin on average, so even small conversion lifts are meaningful.
The team wants to run a controlled A/B test (50/50 traffic split) and needs to decide how long to run the experiment to reliably detect a meaningful improvement.
You are asked to determine the minimum sample size per group required to detect a lift in checkout conversion rate with adequate power, and then translate that into estimated test duration given expected traffic.
| Item | Value |
|---|---|
| Baseline checkout conversion rate (control) | 0.118 (11.8%) |
| Minimum detectable effect (absolute lift) | 0.006 (0.6 percentage points) |
| Significance level | α = 0.05 |
| Desired power | 1 − β = 0.80 |
| Allocation | 50% control / 50% treatment |
| Eligible checkout-start users per day | 320,000 |
| Guardrail: you will not ship unless the 95% CI excludes 0 | Use two-sided test |