Sample Size for Checkout A/B Test | Dataford Interview Questions

Business Context

CartJet is a large e-commerce marketplace (~8M weekly active users) optimizing its mobile checkout. A redesign reduces the number of form fields and adds Apple Pay earlier in the flow. Finance estimates that each additional completed order is worth $6.40 in contribution margin on average, so even small conversion lifts are meaningful.

The team wants to run a controlled A/B test (50/50 traffic split) and needs to decide how long to run the experiment to reliably detect a meaningful improvement.

Problem Statement

You are asked to determine the minimum sample size per group required to detect a lift in checkout conversion rate with adequate power, and then translate that into estimated test duration given expected traffic.

Given Data

Item	Value
Baseline checkout conversion rate (control)	0.118 (11.8%)
Minimum detectable effect (absolute lift)	0.006 (0.6 percentage points)
Significance level	α = 0.05
Desired power	1 − β = 0.80
Allocation	50% control / 50% treatment
Eligible checkout-start users per day	320,000
Guardrail: you will not ship unless the 95% CI excludes 0	Use two-sided test

Requirements

State the null and alternative hypotheses for the primary metric (conversion rate).
Compute the minimum required sample size per group using a normal approximation for a two-sample test of proportions.
Convert the sample size into an estimated test duration in days given the daily eligible traffic and 50/50 split.
Briefly explain what inputs most strongly drive sample size (and how product decisions like choosing a smaller MDE affect runtime).

Assumptions and Constraints

Users are randomly assigned and each user contributes at most one checkout attempt (independent Bernoulli outcomes).
Use a two-sided hypothesis test at α = 0.05.
Use the common planning approximation for two proportions:
- Use p̄ = (p1 + p2)/2 in the variance term (planning value).
- z-values from the standard normal distribution.
Ignore sequential testing/peeking adjustments (assume you will not stop early based on interim looks).

Business Context

The team wants to run a controlled A/B test (50/50 traffic split) and needs to decide how long to run the experiment to reliably detect a meaningful improvement.

Given Data

Item	Value
Baseline checkout conversion rate (control)	0.118 (11.8%)
Minimum detectable effect (absolute lift)	0.006 (0.6 percentage points)
Significance level	α = 0.05
Desired power	1 − β = 0.80
Allocation	50% control / 50% treatment
Eligible checkout-start users per day	320,000
Guardrail: you will not ship unless the 95% CI excludes 0	Use two-sided test

Requirements

State the null and alternative hypotheses for the primary metric (conversion rate).

Compute the minimum required sample size per group using a normal approximation for a two-sample test of proportions.

Convert the sample size into an estimated test duration in days given the daily eligible traffic and 50/50 split.

Briefly explain what inputs most strongly drive sample size (and how product decisions like choosing a smaller MDE affect runtime).

Assumptions and Constraints

Users are randomly assigned and each user contributes at most one checkout attempt (independent Bernoulli outcomes).

Use a two-sided hypothesis test at α = 0.05.

Use the common planning approximation for two proportions:

Use p̄ = (p1 + p2)/2 in the variance term (planning value).
z-values from the standard normal distribution.

Ignore sequential testing/peeking adjustments (assume you will not stop early based on interim looks).