Context
Lyft is considering a new in-app ad placement on the rider home screen that appears before a user requests a ride. The ads team believes the placement can increase ad engagement, but the rider team is concerned it could distract users and hurt the core ride-request experience.
Hypothesis Seed
The proposed change adds a sponsored card above the usual destination-entry module. Product expects higher ad click-through and downstream advertiser value, but only if rider intent metrics such as ride-request conversion and booking completion are not materially harmed.
Constraints
- Eligible traffic: 1.2M rider app sessions per day in the U.S.
- About 55% of sessions reach the home screen and are eligible for exposure
- Maximum experiment duration: 14 days, because ad sales wants a launch decision before the next planning cycle
- False positives are costly: shipping a harmful ad experience could reduce completed rides and rider trust
- False negatives are also meaningful: the ads team estimates roughly $1.5M annualized upside if the placement truly improves engagement
- The experiment must cover weekday/weekend behavior and support a clear ship/no-ship decision
Task
Design a rigorous A/B test for this decision.
- Define the null and alternative hypotheses, including whether you would use a one-sided or two-sided test.
- Specify the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Be explicit about metric definitions, unit of analysis, baseline assumptions, and the minimum detectable effect (MDE).
- Calculate the required sample size and estimate whether the test can be completed within 14 days using the available traffic. Show the math and state all assumptions.
- Choose the unit of randomization, allocation strategy, duration, and any stratification or ramp plan. Explain how you would handle repeated sessions from the same rider.
- Pre-register an analysis plan: statistical test, peeking policy, multiple-comparisons treatment, and what you would do about common pitfalls such as novelty effects, network interference, SUTVA violations, and sample ratio mismatch.
Your final answer should end with a clear ship / don’t ship / iterate rule that respects both the primary metric and the guardrails.
