Sample Size for Checkout Test | Dataford Interview Questions

Context

ShopNow, a mobile e-commerce app, wants to test a simplified checkout flow that removes one confirmation step. Leadership needs a decision this month because the change affects the holiday release train.

Hypothesis Seed

The product team believes the shorter flow will increase purchase conversion by reducing friction at checkout. However, a false positive is costly because shipping a broken flow could reduce revenue and increase payment failures; a false negative is also meaningful because delaying a real improvement leaves revenue on the table.

Constraints

Eligible traffic: 120,000 checkout-starting users per day
Current checkout-to-purchase conversion: 24%
Maximum experiment duration: 14 days
Standard significance level: 5%
Desired power: 80%
Engineering asks for a recommendation only if the test can detect at least a 2% relative lift in conversion within the time window
The experiment platform supports user-level randomization and daily monitoring for safety only

Task

Define the null and alternative hypotheses, the primary metric, 2-4 guardrail metrics, and the minimum detectable effect (MDE). Be explicit about the unit of randomization and unit of analysis.
Calculate the required sample size per arm using the baseline conversion, alpha, power, and MDE above. Show the formula, plug in the numbers, and convert the result into an estimated runtime given available traffic.
Propose the experiment design: allocation, duration, any stratification, and why your randomization choice avoids contamination.
Pre-register the analysis plan: statistical test, peeking policy, how you will handle multiple comparisons, and what you will do if you detect sample ratio mismatch or instrumentation issues.
State a clear ship / don’t ship / iterate rule that respects guardrails. Also explain how you would interpret a statistically significant result that is smaller than the pre-specified MDE.

Assume no major seasonality shocks during the 14-day window, but note any risks that could still bias interpretation.

Problem

Context

Hypothesis Seed

Constraints

Eligible traffic: 120,000 checkout-starting users per day
Current checkout-to-purchase conversion: 24%
Maximum experiment duration: 14 days
Standard significance level: 5%
Desired power: 80%
Engineering asks for a recommendation only if the test can detect at least a 2% relative lift in conversion within the time window
The experiment platform supports user-level randomization and daily monitoring for safety only

Task

Define the null and alternative hypotheses, the primary metric, 2-4 guardrail metrics, and the minimum detectable effect (MDE). Be explicit about the unit of randomization and unit of analysis.
Calculate the required sample size per arm using the baseline conversion, alpha, power, and MDE above. Show the formula, plug in the numbers, and convert the result into an estimated runtime given available traffic.
Propose the experiment design: allocation, duration, any stratification, and why your randomization choice avoids contamination.
Pre-register the analysis plan: statistical test, peeking policy, how you will handle multiple comparisons, and what you will do if you detect sample ratio mismatch or instrumentation issues.
State a clear ship / don’t ship / iterate rule that respects guardrails. Also explain how you would interpret a statistically significant result that is smaller than the pre-specified MDE.

Assume no major seasonality shocks during the 14-day window, but note any risks that could still bias interpretation.

Problem

Context

Hypothesis Seed

Constraints

Eligible traffic: 120,000 checkout-starting users per day
Current checkout-to-purchase conversion: 24%
Maximum experiment duration: 14 days
Standard significance level: 5%
Desired power: 80%
Engineering asks for a recommendation only if the test can detect at least a 2% relative lift in conversion within the time window
The experiment platform supports user-level randomization and daily monitoring for safety only

Task

Define the null and alternative hypotheses, the primary metric, 2-4 guardrail metrics, and the minimum detectable effect (MDE). Be explicit about the unit of randomization and unit of analysis.
Calculate the required sample size per arm using the baseline conversion, alpha, power, and MDE above. Show the formula, plug in the numbers, and convert the result into an estimated runtime given available traffic.
Propose the experiment design: allocation, duration, any stratification, and why your randomization choice avoids contamination.
Pre-register the analysis plan: statistical test, peeking policy, how you will handle multiple comparisons, and what you will do if you detect sample ratio mismatch or instrumentation issues.
State a clear ship / don’t ship / iterate rule that respects guardrails. Also explain how you would interpret a statistically significant result that is smaller than the pre-specified MDE.

Assume no major seasonality shocks during the 14-day window, but note any risks that could still bias interpretation.

Problem

Context

Hypothesis Seed

Constraints

Eligible traffic: 120,000 checkout-starting users per day
Current checkout-to-purchase conversion: 24%
Maximum experiment duration: 14 days
Standard significance level: 5%
Desired power: 80%
Engineering asks for a recommendation only if the test can detect at least a 2% relative lift in conversion within the time window
The experiment platform supports user-level randomization and daily monitoring for safety only

Task

Define the null and alternative hypotheses, the primary metric, 2-4 guardrail metrics, and the minimum detectable effect (MDE). Be explicit about the unit of randomization and unit of analysis.
Calculate the required sample size per arm using the baseline conversion, alpha, power, and MDE above. Show the formula, plug in the numbers, and convert the result into an estimated runtime given available traffic.
Propose the experiment design: allocation, duration, any stratification, and why your randomization choice avoids contamination.
Pre-register the analysis plan: statistical test, peeking policy, how you will handle multiple comparisons, and what you will do if you detect sample ratio mismatch or instrumentation issues.
State a clear ship / don’t ship / iterate rule that respects guardrails. Also explain how you would interpret a statistically significant result that is smaller than the pre-specified MDE.

Assume no major seasonality shocks during the 14-day window, but note any risks that could still bias interpretation.