Explaining Statistical Power for Email Tests | Dataford Interview Questions

Your question is Explaining Statistical Power for Email Tests. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Business Context

BrightMail, a SaaS email platform, is planning an A/B test on a new onboarding email subject line. The marketing director is non-technical and asks why the data team wants a larger sample before concluding that the new subject line does not work.

Problem Statement

Use the test setup below to explain statistical power in plain business terms and quantify what it means for this experiment. Show how likely the team is to detect a real improvement if one exists.

Given Data

Metric	Value
Baseline open rate	20.0%
Expected treatment open rate	22.0%
Absolute lift to detect	2.0 percentage points
Significance level	5%
Test type	Two-sided two-proportion z-test
Planned sample per group	2,500
Alternative sample per group	8,000

Assume equal traffic split between control and treatment.

Requirements

Define statistical power in non-technical language.
State the null and alternative hypotheses for this test.
Compute the standard error under the null and the rejection threshold at $\alpha = 0.05$ .
Approximate the power when the true open rates are 20.0% vs 22.0% with 2,500 users per group.
Approximate the power again with 8,000 users per group.
Explain what the results imply for business decision-making and why a non-significant result from the smaller test would be hard to interpret.

Assumptions

Users are randomly assigned and counted once.
The normal approximation is appropriate for the sample sizes used.
Ignore multiple testing and operational issues such as deliverability changes.