Business Context
Lyft is testing a new in-app ad product in the Lyft rider app, shown on the post-ride receipt screen. The team wants to know whether the ad unit increases monetization enough to justify any degradation in rider experience.
Problem Statement
Design and analyze an experiment to estimate the causal impact of the new ad product on ad revenue per exposed rider while protecting user experience. Assume Lyft randomized eligible riders 50/50 into control (no ad on receipt screen) and treatment (new ad shown).
Given Data
| Metric | Control | Treatment |
|---|
| Eligible riders | 120,000 | 120,000 |
| Riders with at least 1 completed ride | 96,400 | 96,100 |
| Total 7-day ad revenue | \$0 | \$33,154.50 |
| Mean 7-day total revenue per eligible rider (rides + ads) | \$18.420 | \$18.696 |
| Std. dev. of 7-day total revenue per eligible rider | \$12.10 | \$12.40 |
| Riders opening the app again within 7 days | 42,388 | 40,842 |
Additional assumptions for planning:
- Baseline 7-day re-open rate: 44.0%
- Minimum detectable decline in re-open rate worth catching: 1.5 percentage points
- Significance level: 0.05
- Desired power: 80%
Requirements
- Define a primary metric and at least one guardrail metric.
- State null and alternative hypotheses for both monetization and user experience.
- Test whether treatment increased 7-day total revenue per eligible rider.
- Test whether treatment reduced 7-day re-open rate.
- Compute a 95% confidence interval for the revenue lift and for the re-open-rate difference.
- Estimate the required sample size per group to detect a 1.5 percentage point drop in re-open rate.
- Give a launch recommendation and note key experimental risks.
Assumptions
- Randomization is at the eligible rider level and stable over the 7-day window.
- Revenue per rider is approximately normal by the central limit theorem at this sample size.
- Re-open behavior is binary and independent across riders.
