Minimum Detectable Effect for Signup Test

Business Context

StreamCart is planning an A/B test on a new signup page. Before launch, the product team wants to know the smallest conversion lift the experiment can reliably detect with the traffic available over a 3-week test window.

Problem Statement

Determine the minimum detectable effect (MDE) for a two-sample test of proportions, assuming a two-sided hypothesis test, 5% significance level, and 80% power. Then convert that MDE into the minimum detectable number of additional signups.

Given Data

This means each variant will receive 120,000 visitors if the test runs as planned.

Requirements

1. State the hypothesis-testing setup used for MDE planning.
2. Compute the per-group sample size.
3. Use the normal approximation for two independent proportions to calculate the absolute MDE in percentage points.
4. Convert the absolute MDE into a relative lift versus baseline.
5. Estimate the minimum additional signups needed in treatment for the effect to be detectable at the planned sample size.
6. Briefly explain how MDE would change if traffic were lower or if the team wanted 90% power instead of 80%.

Assumptions

• Random assignment is valid and independent across users.
• Signup is a binary outcome measured once per visitor.
• Baseline conversion is stable during the test window.
• Use the standard planning approximation with equal-sized groups and variance evaluated at the baseline rate.