Context
StayNest, a vacation-rental marketplace, redesigned its checkout flow to reduce friction and increase completed bookings. An initial readout shows a 5% relative increase in bookings but also a 2% relative increase in cancellations, and the GM asks whether you would launch.
Hypothesis Seed
The new flow removes one confirmation step and pre-fills traveler details. The product team believes this will improve booking conversion by making checkout faster, but there is concern that easier checkout may attract lower-intent bookings that later cancel.
Constraints
- Eligible traffic: 120,000 checkout-starting users per day
- Baseline booking rate from checkout start: 12.0%
- Baseline cancellation rate among booked reservations within 7 days: 18.0%
- Maximum experiment runtime: 21 days
- Decision deadline: launch, iterate, or stop at the end of the test window
- False positives are costly because cancellations create host support costs and reduce marketplace trust; false negatives are also costly because booking growth is a top quarterly goal
- Assume 50/50 allocation and that the team can tolerate at most a 1% relative increase in cancellations if booking conversion improves materially
Task
- Define the experiment clearly: hypothesis, primary metric, guardrails, secondary metrics, unit of randomization, and MDE.
- Calculate the required sample size and expected duration using explicit assumptions for α = 0.05 and 80% power. Show the math with real numbers.
- Write a pre-registered analysis plan: statistical test, handling of multiple metrics, peeking policy, and the exact ship / don’t-ship rule.
- Explain how you would interpret a result where bookings are up 5% but cancellations are also up 2%, including the difference between statistical and practical significance.
- Identify key experimentation risks such as novelty effects, sample ratio mismatch, and interference / SUTVA concerns, and explain how you would mitigate them.
