Interview Guides

Sample Size for Signup Experiment

Medium

Statistics & Probability

A/B TestingPower AnalysisSample Size

Problem

Business Context

StreamCart, a subscription video platform, wants to test a simplified signup page. The team does not want to launch the experiment until they know how many users are needed to reliably detect a meaningful lift in signup conversion.

Problem Statement

Determine the minimum sample size per group required for a two-arm A/B test comparing conversion rates. The team wants enough power to detect a small but commercially meaningful improvement from the current baseline.

Given Data

Metric	Value
Baseline conversion rate $p_1$	8.2%
Minimum detectable effect (absolute lift)	1.0 percentage point
Expected treatment conversion rate $p_2$	9.2%
Significance level $\alpha$	0.05
Desired power $1-\beta$	80%
Test type	Two-sided
Traffic split	50/50

Assume the product team will use a normal approximation for two independent proportions.

Requirements

State the null and alternative hypotheses.
Compute the critical values for $\alpha$ and power.
Calculate the required sample size per variant for a two-proportion A/B test.
Calculate the total required sample size.
If StreamCart receives 52,000 eligible users per day and only 60% can be exposed to the experiment, estimate the test duration.
Briefly explain how the required sample size would change if the MDE were smaller or if the team wanted 90% power instead of 80%.

Assumptions

Users are independently randomized and counted once.
Conversion is binary at the user level.
No peeking or early stopping adjustments are planned.
The baseline rate estimate of 8.2% is stable enough for planning purposes.

Sample Data

Example 1

Input

{"power":0.8,"absolute_mde":0.01,"significance_level":0.05,"daily_eligible_users":52000,"allocation_to_treatment":0.5,"baseline_conversion_rate":0.082,"treatment_conversion_rate":0.092,"experiment_traffic_fraction":0.6}

Output(none)

Problem

Business Context

Problem Statement

Given Data

Metric	Value
Baseline conversion rate $p_1$	8.2%
Minimum detectable effect (absolute lift)	1.0 percentage point
Expected treatment conversion rate $p_2$	9.2%
Significance level $\alpha$	0.05
Desired power $1-\beta$	80%
Test type	Two-sided
Traffic split	50/50

Assume the product team will use a normal approximation for two independent proportions.

Requirements

State the null and alternative hypotheses.
Compute the critical values for $\alpha$ and power.
Calculate the required sample size per variant for a two-proportion A/B test.
Calculate the total required sample size.
If StreamCart receives 52,000 eligible users per day and only 60% can be exposed to the experiment, estimate the test duration.
Briefly explain how the required sample size would change if the MDE were smaller or if the team wanted 90% power instead of 80%.

Assumptions

Users are independently randomized and counted once.
Conversion is binary at the user level.
No peeking or early stopping adjustments are planned.
The baseline rate estimate of 8.2% is stable enough for planning purposes.

Sample Data

Example 1

Input

{"power":0.8,"absolute_mde":0.01,"significance_level":0.05,"daily_eligible_users":52000,"allocation_to_treatment":0.5,"baseline_conversion_rate":0.082,"treatment_conversion_rate":0.092,"experiment_traffic_fraction":0.6}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Sample Size Planning for Signup TestMedium

Minimum Detectable Effect for Signup TestMedium Sample Size for Signup Growth TestMedium

Next question

Sample Size for Signup Experiment

Medium

Statistics & Probability

A/B TestingPower AnalysisSample Size

Problem

Business Context

Problem Statement

Given Data

Metric	Value
Baseline conversion rate $p_1$	8.2%
Minimum detectable effect (absolute lift)	1.0 percentage point
Expected treatment conversion rate $p_2$	9.2%
Significance level $\alpha$	0.05
Desired power $1-\beta$	80%
Test type	Two-sided
Traffic split	50/50

Assume the product team will use a normal approximation for two independent proportions.

Requirements

State the null and alternative hypotheses.
Compute the critical values for $\alpha$ and power.
Calculate the required sample size per variant for a two-proportion A/B test.
Calculate the total required sample size.
If StreamCart receives 52,000 eligible users per day and only 60% can be exposed to the experiment, estimate the test duration.
Briefly explain how the required sample size would change if the MDE were smaller or if the team wanted 90% power instead of 80%.

Assumptions

Users are independently randomized and counted once.
Conversion is binary at the user level.
No peeking or early stopping adjustments are planned.
The baseline rate estimate of 8.2% is stable enough for planning purposes.

Sample Data

Example 1

Input

{"power":0.8,"absolute_mde":0.01,"significance_level":0.05,"daily_eligible_users":52000,"allocation_to_treatment":0.5,"baseline_conversion_rate":0.082,"treatment_conversion_rate":0.092,"experiment_traffic_fraction":0.6}

Output(none)

Problem

Business Context

Problem Statement

Given Data

Metric	Value
Baseline conversion rate $p_1$	8.2%
Minimum detectable effect (absolute lift)	1.0 percentage point
Expected treatment conversion rate $p_2$	9.2%
Significance level $\alpha$	0.05
Desired power $1-\beta$	80%
Test type	Two-sided
Traffic split	50/50

Assume the product team will use a normal approximation for two independent proportions.

Requirements

State the null and alternative hypotheses.
Compute the critical values for $\alpha$ and power.
Calculate the required sample size per variant for a two-proportion A/B test.
Calculate the total required sample size.
If StreamCart receives 52,000 eligible users per day and only 60% can be exposed to the experiment, estimate the test duration.
Briefly explain how the required sample size would change if the MDE were smaller or if the team wanted 90% power instead of 80%.

Assumptions

Users are independently randomized and counted once.
Conversion is binary at the user level.
No peeking or early stopping adjustments are planned.
The baseline rate estimate of 8.2% is stable enough for planning purposes.

Sample Data

Example 1

Input

{"power":0.8,"absolute_mde":0.01,"significance_level":0.05,"daily_eligible_users":52000,"allocation_to_treatment":0.5,"baseline_conversion_rate":0.082,"treatment_conversion_rate":0.092,"experiment_traffic_fraction":0.6}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

Sample Size Planning for Signup TestMedium

Minimum Detectable Effect for Signup TestMedium Sample Size for Signup Growth TestMedium

Next question