Business Context
StreamCart wants to test a new signup page that may improve visitor-to-account conversion. Before launching the experiment, the product team wants to know how many users are needed to reliably detect a meaningful lift.
Problem Statement
Plan the sample size for a two-arm A/B test on conversion rate. The current signup conversion rate is 18.0%, and the team only wants to run the test if it has enough power to detect an absolute lift of at least 1.5 percentage points.
Given Data
| Metric | Value |
|---|
| Baseline conversion rate | 18.0% |
| Minimum detectable effect (absolute) | 1.5 percentage points |
| Expected treatment conversion rate | 19.5% |
| Significance level | 0.05 |
| Desired power | 80% |
| Test type | Two-sided |
| Traffic split | 50/50 |
| Weekly eligible visitors | 120,000 |
Requirements
- State the null and alternative hypotheses.
- Compute the required sample size per group for a two-proportion test using the normal approximation.
- Compute the total sample size and estimated test duration in weeks.
- If the team can only collect 8,000 users per group, estimate the approximate power for detecting the same 1.5 percentage point lift.
- Explain how sample size, effect size, significance level, and power trade off in experiment planning.
Assumptions
- Users are independently and randomly assigned.
- The baseline rate of 18.0% is stable during the test window.
- The normal approximation is appropriate because expected successes and failures are large in both groups.
- Ignore multiple testing and sequential peeking for this calculation.
