Context
ShopNow is testing a redesigned checkout page intended to reduce friction and increase completed purchases. Because checkout is a critical path, the team wants an experiment design that explicitly protects overall system performance while still measuring business impact.
Hypothesis Seed
The new checkout removes one form step and prefetches payment options earlier in the flow. Product expects this to improve purchase conversion, but engineering is concerned that extra client-side requests could increase page latency, backend load, and payment failures.
Constraints
- Eligible traffic: 600,000 checkout sessions per day
- Maximum experiment duration: 14 days
- Allocation can be ramped, but final steady-state split should not exceed 50/50
- Baseline checkout completion rate: 48%
- The smallest business-relevant lift is 1.5% relative in checkout completion
- A false positive is expensive because a bad launch can degrade revenue and site reliability during peak traffic
- A false negative is acceptable if it avoids shipping a risky experience
Deliverables
- Define the experiment hypothesis, the primary success metric, and 2-4 system-performance guardrails that would prevent shipping a harmful variant.
- Calculate the required sample size for the primary metric using an explicit MDE, and estimate whether the test can finish within the 14-day limit.
- Choose the unit of randomization, allocation plan, and duration; explain how your design avoids contamination and captures weekly traffic patterns.
- Pre-register the analysis plan: statistical test, peeking policy, treatment of guardrails, and how you will handle any mismatch between unit of randomization and unit of analysis.
- State a clear ship / don’t-ship / iterate rule that respects both the primary metric and guardrails, and identify key pitfalls that could invalidate the result.
