StreamCart, a subscription video platform, tested a new homepage signup CTA. After 10 days, the product manager wants to ship because treatment looks better on the primary metric.
You need to determine whether the observed lift in signup conversion is statistically significant and whether the experiment itself may be misleading due to common A/B testing pitfalls.
| Metric | Control | Treatment |
|---|---|---|
| Assigned users | 52,400 | 47,600 |
| Signups | 6,026 | 5,950 |
| Observed conversion rate | 11.50% | 12.50% |
| Expected traffic split | 50% | 50% |
Additional experiment facts:
| Item | Value |
|---|---|
| Test duration | 10 days |
| Significance level | 0.05 |
| Number of dashboard checks during run | 8 |
| Other metrics reviewed post hoc | 6 |
| Day 1 treatment conversion rate | 14.2% |
| Days 2-10 treatment conversion rate | 12.3% |