Context
ShopNow, a mobile commerce app, wants to simplify its checkout flow by replacing a 3-step checkout with a new 1-page checkout. Leadership wants evidence that the change improves conversion without hurting order quality or support burden.
Hypothesis Seed
The product team believes reducing friction in checkout will increase purchase conversion among users who start checkout. However, they are concerned that a faster flow could increase accidental purchases, payment failures, or refund requests.
Constraints
- Eligible traffic: 120,000 checkout-starting users per day
- Maximum experiment window: 14 days, after which the team must make a launch decision for the next release cycle
- Allocation can be 50/50 after a brief instrumentation ramp
- Baseline checkout conversion rate (purchase within 24 hours of checkout start): 32%
- The smallest business-meaningful lift is +2.0% relative in checkout conversion
- False positives are costly because a bad launch would affect revenue and customer trust; false negatives are acceptable if the team can iterate next sprint
Deliverables
- Define the null and alternative hypotheses, the primary metric, 2-4 guardrail metrics, and at least 1 secondary metric. Be explicit about the metric formula, unit of analysis, and MDE.
- Calculate the required sample size per arm using the baseline conversion rate, =0.05, and 80% power. Translate that into expected runtime given the available traffic.
- Choose the unit of randomization, allocation plan, duration, and any stratification or blocking you would use. Explain why your design avoids contamination.
- Pre-register an analysis plan: statistical test, peeking policy, multiple-comparison policy, and how you will handle any mismatch between unit of randomization and unit of analysis.
- State a clear ship / don’t ship / iterate rule, and identify key risks such as novelty effects, sample ratio mismatch, or interference across users/devices.
