Business Context
StreamCart is planning an A/B test on a new signup page. Before launch, the product team wants to know the smallest conversion lift the experiment can reliably detect with the traffic available over a 3-week test window.
Problem Statement
Determine the minimum detectable effect (MDE) for a two-sample test of proportions, assuming a two-sided hypothesis test, 5% significance level, and 80% power. Then convert that MDE into the minimum detectable number of additional signups.
Given Data
| Metric | Value |
|---|
| Baseline signup conversion rate | 8.4% |
| Expected eligible visitors during test | 240,000 |
| Traffic split | 50% control / 50% treatment |
| Significance level | 0.05 |
| Desired power | 0.80 |
| Test type | Two-sided |
This means each variant will receive 120,000 visitors if the test runs as planned.
Requirements
- State the hypothesis-testing setup used for MDE planning.
- Compute the per-group sample size.
- Use the normal approximation for two independent proportions to calculate the absolute MDE in percentage points.
- Convert the absolute MDE into a relative lift versus baseline.
- Estimate the minimum additional signups needed in treatment for the effect to be detectable at the planned sample size.
- Briefly explain how MDE would change if traffic were lower or if the team wanted 90% power instead of 80%.
Assumptions
- Random assignment is valid and independent across users.
- Signup is a binary outcome measured once per visitor.
- Baseline conversion is stable during the test window.
- Use the standard planning approximation with equal-sized groups and variance evaluated at the baseline rate.
