Experiment Runtime for Signup Test | Dataford Interview Questions

Business Context

StreamCart is testing a new signup page intended to increase free-trial starts. The product manager wants to know how long the experiment must run before the team can reasonably expect to detect a meaningful lift with statistical significance.

Problem Statement

Estimate the required sample size and runtime for a two-arm A/B test on conversion rate, then check whether the currently observed data are already statistically significant.

Given Data

Metric	Value
Baseline signup conversion rate	8.2%
Minimum detectable effect (relative lift)	10.0%
Significance level	5.0%
Desired power	80.0%
Daily eligible users	52,000
Traffic split	50% control / 50% treatment
Current runtime	10 days
Control users so far	260,000
Treatment users so far	260,000
Control conversions so far	21,320
Treatment conversions so far	23,140

The business defines success as detecting at least a 10% relative lift over the baseline conversion rate.

Requirements

Convert the relative MDE into an absolute treatment conversion rate.
Compute the required sample size per group for a two-sided two-proportion test.
Convert that sample size into the minimum experiment runtime in days.
Using the current 10-day data, run a two-proportion z-test.
Compute a 95% confidence interval for the observed lift.
State whether the test should stop now or continue.

Assumptions

Users are independently randomized at the user level.
Conversion is binary and each user is counted once.
Daily traffic is stable and evenly split.
Normal approximation is appropriate because sample sizes are large.

Problem

Business Context

Problem Statement

Estimate the required sample size and runtime for a two-arm A/B test on conversion rate, then check whether the currently observed data are already statistically significant.

Given Data

Metric	Value
Baseline signup conversion rate	8.2%
Minimum detectable effect (relative lift)	10.0%
Significance level	5.0%
Desired power	80.0%
Daily eligible users	52,000
Traffic split	50% control / 50% treatment
Current runtime	10 days
Control users so far	260,000
Treatment users so far	260,000
Control conversions so far	21,320
Treatment conversions so far	23,140

The business defines success as detecting at least a 10% relative lift over the baseline conversion rate.

Requirements

Convert the relative MDE into an absolute treatment conversion rate.
Compute the required sample size per group for a two-sided two-proportion test.
Convert that sample size into the minimum experiment runtime in days.
Using the current 10-day data, run a two-proportion z-test.
Compute a 95% confidence interval for the observed lift.
State whether the test should stop now or continue.

Assumptions

Users are independently randomized at the user level.
Conversion is binary and each user is counted once.
Daily traffic is stable and evenly split.
Normal approximation is appropriate because sample sizes are large.

Problem

Business Context

Problem Statement

Estimate the required sample size and runtime for a two-arm A/B test on conversion rate, then check whether the currently observed data are already statistically significant.

Given Data

Metric	Value
Baseline signup conversion rate	8.2%
Minimum detectable effect (relative lift)	10.0%
Significance level	5.0%
Desired power	80.0%
Daily eligible users	52,000
Traffic split	50% control / 50% treatment
Current runtime	10 days
Control users so far	260,000
Treatment users so far	260,000
Control conversions so far	21,320
Treatment conversions so far	23,140

The business defines success as detecting at least a 10% relative lift over the baseline conversion rate.

Requirements

Convert the relative MDE into an absolute treatment conversion rate.
Compute the required sample size per group for a two-sided two-proportion test.
Convert that sample size into the minimum experiment runtime in days.
Using the current 10-day data, run a two-proportion z-test.
Compute a 95% confidence interval for the observed lift.
State whether the test should stop now or continue.

Assumptions

Users are independently randomized at the user level.
Conversion is binary and each user is counted once.
Daily traffic is stable and evenly split.
Normal approximation is appropriate because sample sizes are large.

Problem

Business Context

Problem Statement

Estimate the required sample size and runtime for a two-arm A/B test on conversion rate, then check whether the currently observed data are already statistically significant.

Given Data

Metric	Value
Baseline signup conversion rate	8.2%
Minimum detectable effect (relative lift)	10.0%
Significance level	5.0%
Desired power	80.0%
Daily eligible users	52,000
Traffic split	50% control / 50% treatment
Current runtime	10 days
Control users so far	260,000
Treatment users so far	260,000
Control conversions so far	21,320
Treatment conversions so far	23,140

The business defines success as detecting at least a 10% relative lift over the baseline conversion rate.

Requirements

Convert the relative MDE into an absolute treatment conversion rate.
Compute the required sample size per group for a two-sided two-proportion test.
Convert that sample size into the minimum experiment runtime in days.
Using the current 10-day data, run a two-proportion z-test.
Compute a 95% confidence interval for the observed lift.
State whether the test should stop now or continue.

Assumptions

Users are independently randomized at the user level.
Conversion is binary and each user is counted once.
Daily traffic is stable and evenly split.
Normal approximation is appropriate because sample sizes are large.