Business Context

BrightMail, a SaaS email platform, is planning an A/B test on a new onboarding email subject line. The marketing director is non-technical and asks why the data team wants a larger sample before concluding that the new subject line does not work.

Problem Statement

Use the test setup below to explain statistical power in plain business terms and quantify what it means for this experiment. Show how likely the team is to detect a real improvement if one exists.

Given Data

Assume equal traffic split between control and treatment.

Requirements

1. Define statistical power in non-technical language.
2. State the null and alternative hypotheses for this test.
3. Compute the standard error under the null and the rejection threshold at $\alpha = 0.05$.
4. Approximate the power when the true open rates are 20.0% vs 22.0% with 2,500 users per group.
5. Approximate the power again with 8,000 users per group.
6. Explain what the results imply for business decision-making and why a non-significant result from the smaller test would be hard to interpret.

Assumptions

• Users are randomly assigned and counted once.
• The normal approximation is appropriate for the sample sizes used.
• Ignore multiple testing and operational issues such as deliverability changes.