





BrightMail, a SaaS email platform, is planning an A/B test on a new onboarding email subject line. The marketing director is non-technical and asks why the data team wants a larger sample before concluding that the new subject line does not work.
Use the test setup below to explain statistical power in plain business terms and quantify what it means for this experiment. Show how likely the team is to detect a real improvement if one exists.
| Metric | Value |
|---|---|
| Baseline open rate | 20.0% |
| Expected treatment open rate | 22.0% |
| Absolute lift to detect | 2.0 percentage points |
| Significance level | 5% |
| Test type | Two-sided two-proportion z-test |
| Planned sample per group | 2,500 |
| Alternative sample per group | 8,000 |
Assume equal traffic split between control and treatment.