Context
ShopLane, an e-commerce retailer, is considering offering free standard shipping on eligible orders to increase conversion. Leadership will only launch if the change improves overall profitability, not just order volume.
Hypothesis Seed
The product team believes removing the shipping fee at checkout will reduce purchase friction, increase conversion rate, and raise total contribution profit despite higher shipping subsidy costs. Finance is concerned the policy may attract low-margin orders and reduce profit per visitor.
Constraints
- Eligible traffic: 120,000 checkout-starting users per day
- Average baseline checkout conversion: 8.0%
- Baseline contribution profit per checkout-starting user: $1.60
- Standard deviation of contribution profit per user: $12.00
- Maximum experiment duration: 21 days
- Shipping subsidy would cost an incremental $3.20 per converted treatment order on average
- False positive cost is high: shipping an unprofitable policy would cost millions annually
- False negative cost is moderate: delaying a good policy by a few weeks is acceptable
Task
- Define the null and alternative hypotheses, the primary metric, 2-4 guardrails, and a minimum detectable effect (MDE) that is meaningful for the business.
- Calculate the required sample size and determine whether the experiment can be completed within 21 days given the available traffic.
- Choose the unit of randomization, allocation, duration, and any stratification or blocking strategy.
- Pre-register the analysis plan: statistical test, peeking policy, treatment of multiple comparisons, and how you will handle any mismatch between unit of randomization and unit of analysis.
- State a clear ship / don’t-ship / iterate rule that respects guardrails and explain the main experimental risks.
