Context
Quora is testing a change to the Home Feed ranking on logged-in users: the treatment shows more immediately engaging answers and follow-up questions near the top of feed. Early readouts suggest higher engagement, but the team is worried this may reduce long-term retention if users feel the feed becomes lower quality or more addictive than useful.
Hypothesis Seed
The growth team believes the new Home Feed ranking will increase short-term engagement by making content more clickable and session-worthy. However, the key question is whether that engagement is durable value creation or whether it trades off against medium-term user retention.
Constraints
- Eligible traffic: 1.2M logged-in Home Feed users per day
- Maximum experiment duration: 28 days of exposure, with a decision due within 6 weeks
- 50/50 allocation after a 1-day ramp
- Baseline 28-day retained-user rate: 36%
- Baseline engaged sessions per user in first 7 days: 5.2, standard deviation 8.0
- False positive cost is high: shipping a feed that harms retention would be expensive and slow to reverse
- False negative cost is moderate: delaying a good ranking change by a few weeks is acceptable
Deliverables
- Define a clear hypothesis, including whether the primary decision metric should be engagement or retention, and specify primary, secondary, and guardrail metrics for Quora Home Feed.
- Calculate the required sample size and minimum detectable effect for the primary metric, then translate that into expected runtime given the available traffic.
- Choose the unit of randomization and explain how you would handle the fact that engagement is observed quickly while retention is a lagging outcome.
- Pre-register an analysis plan: statistical test, peeking policy, multiple-comparison policy, and how you would diagnose an apparent engagement-up / retention-down result.
- State a ship / don’t-ship / iterate rule that explicitly respects guardrails and explains what you would do if engagement is significantly up but retention is significantly down.
