Context
ShopNow, a mobile e-commerce app, wants to test a simplified checkout flow that removes one confirmation step. Leadership needs a decision this month because the change affects the holiday release train.
Hypothesis Seed
The product team believes the shorter flow will increase purchase conversion by reducing friction at checkout. However, a false positive is costly because shipping a broken flow could reduce revenue and increase payment failures; a false negative is also meaningful because delaying a real improvement leaves revenue on the table.
Constraints
- Eligible traffic: 120,000 checkout-starting users per day
- Current checkout-to-purchase conversion: 24%
- Maximum experiment duration: 14 days
- Standard significance level: 5%
- Desired power: 80%
- Engineering asks for a recommendation only if the test can detect at least a 2% relative lift in conversion within the time window
- The experiment platform supports user-level randomization and daily monitoring for safety only
Task
- Define the null and alternative hypotheses, the primary metric, 2-4 guardrail metrics, and the minimum detectable effect (MDE). Be explicit about the unit of randomization and unit of analysis.
- Calculate the required sample size per arm using the baseline conversion, alpha, power, and MDE above. Show the formula, plug in the numbers, and convert the result into an estimated runtime given available traffic.
- Propose the experiment design: allocation, duration, any stratification, and why your randomization choice avoids contamination.
- Pre-register the analysis plan: statistical test, peeking policy, how you will handle multiple comparisons, and what you will do if you detect sample ratio mismatch or instrumentation issues.
- State a clear ship / don’t ship / iterate rule that respects guardrails. Also explain how you would interpret a statistically significant result that is smaller than the pre-specified MDE.
Assume no major seasonality shocks during the 14-day window, but note any risks that could still bias interpretation.
