Context
RideNow, a two-sided ride-hailing marketplace, wants to evaluate a new dispatch algorithm that matches riders to drivers using a richer ETA + driver-utilization objective. The team believes it can reduce rider pickup time and cancellations, but a bad rollout could hurt marketplace balance and driver earnings.
Hypothesis Seed
The new algorithm prioritizes lower predicted pickup ETA while smoothing driver idle time across zones. Product expects faster pickups and fewer rider cancellations, but there is concern that local marketplace interactions may create spillovers across drivers and neighborhoods.
Constraints
- Eligible traffic: 240,000 completed or canceled dispatch requests per day across 12 cities
- Maximum experiment window: 21 days, after which operations needs a ship / no-ship decision
- Business requirement: do not increase rider cancellation rate by more than 0.3 percentage points or reduce driver hourly earnings by more than 1%
- Cost asymmetry: a false positive is expensive because it can degrade marketplace liquidity and trust; a false negative is acceptable if it delays a beneficial launch by a few weeks
- Engineering constraint: the dispatch service can support a city-by-time-cell experiment, but not rider-level randomization without contamination risk
Tasks
- Define the null and alternative hypotheses, the primary metric, at least three guardrails, and a realistic MDE.
- Choose the unit of randomization and experiment design, and justify it given marketplace interference and SUTVA concerns.
- Calculate the required sample size and expected duration using explicit assumptions, including alpha, power, baseline, and MDE.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparison treatment, SRM checks, and how you will handle clustering.
- State a clear ship / don’t-ship / iterate rule that respects guardrails even if the primary metric improves.
