Engagement Lift, Retention Drop

Context

Quora is testing a change to the Home Feed ranking on logged-in users: the treatment shows more immediately engaging answers and follow-up questions near the top of feed. Early readouts suggest higher engagement, but the team is worried this may reduce long-term retention if users feel the feed becomes lower quality or more addictive than useful.

Hypothesis Seed

The growth team believes the new Home Feed ranking will increase short-term engagement by making content more clickable and session-worthy. However, the key question is whether that engagement is durable value creation or whether it trades off against medium-term user retention.

Constraints

• Eligible traffic: 1.2M logged-in Home Feed users per day
• Maximum experiment duration: 28 days of exposure, with a decision due within 6 weeks
• 50/50 allocation after a 1-day ramp
• Baseline 28-day retained-user rate: 36%
• Baseline engaged sessions per user in first 7 days: 5.2, standard deviation 8.0
• False positive cost is high: shipping a feed that harms retention would be expensive and slow to reverse
• False negative cost is moderate: delaying a good ranking change by a few weeks is acceptable

Deliverables

1. Define a clear hypothesis, including whether the primary decision metric should be engagement or retention, and specify primary, secondary, and guardrail metrics for Quora Home Feed.
2. Calculate the required sample size and minimum detectable effect for the primary metric, then translate that into expected runtime given the available traffic.
3. Choose the unit of randomization and explain how you would handle the fact that engagement is observed quickly while retention is a lagging outcome.
4. Pre-register an analysis plan: statistical test, peeking policy, multiple-comparison policy, and how you would diagnose an apparent engagement-up / retention-down result.
5. State a ship / don’t-ship / iterate rule that explicitly respects guardrails and explains what you would do if engagement is significantly up but retention is significantly down.