Context
ShopNow, a mobile commerce app, wants to increase completed purchases from product pages. The growth team proposes a new one-tap checkout banner shown to logged-in users with a saved payment method.
Hypothesis Seed
The team believes surfacing a persistent one-tap checkout banner on eligible product pages will reduce friction and increase purchase conversion. However, leadership is concerned that an aggressive banner could hurt average order value, increase accidental purchases and refunds, or create a short-lived novelty spike.
Constraints
- Eligible traffic: 120,000 product-page visitors per day
- Only 60% of visitors are logged in and eligible for the feature
- Maximum experiment window: 14 days, including at least one full weekend cycle
- Product-page purchase conversion baseline: 8.0%
- The business only wants to ship if the lift is large enough to matter financially; set a clear MDE
- False positives are costly because checkout changes can create trust issues and support burden; false negatives are acceptable if the missed upside is small
- Engineering can support a 50/50 split after a 1-day instrumentation ramp
Task
- Define the experiment clearly: hypothesis, primary metric, 2-4 guardrail metrics, secondary metrics, unit of randomization, and the MDE you would power for.
- Calculate the required sample size per arm using explicit assumptions for alpha, power, baseline rate, and MDE. Then translate that into expected runtime given the available eligible traffic.
- Pre-register an analysis plan: statistical test, treatment of guardrails and secondary metrics, peeking policy, and how you will check for sample ratio mismatch.
- Explain the main risks to valid inference in this test, including novelty effects, unit-of-analysis mistakes, and any interference or SUTVA concerns.
- State a ship / do-not-ship / iterate rule that respects both the primary metric and guardrails.
