Design Facebook Feed A/B Test

Context

The Facebook Feed team wants to test a lightweight ranking change that increases the share of posts from close friends near the top of Feed. Product leadership needs a launch recommendation within three weeks.

Hypothesis Seed

The team believes showing slightly more close-friend content will improve meaningful engagement in Feed because users are more likely to comment and react on posts from people they know well. However, the change could reduce overall session depth or increase negative feedback if users find Feed less diverse.

Constraints

• Eligible traffic: 24M daily active Facebook users globally who open Feed at least once per day
• Maximum experiment duration: 14 days after a 1-day instrumentation ramp
• Allocation target: 50/50 control vs treatment after ramp
• Baseline primary metric: 0.180 meaningful social interactions per daily active Feed user
• Smallest business-relevant lift: 2.0% relative
• False positives are costly because a bad ranking launch affects a large share of Feed impressions; false negatives are acceptable if they avoid shipping a harmful model
• The team wants a clear pre-registered launch rule and does not want ad hoc metric shopping after results arrive

Deliverables

1. State the null and alternative hypotheses, and define the primary metric, secondary metrics, and guardrails.
2. Calculate the required sample size and confirm whether the available traffic can detect the stated MDE within 14 days.
3. Choose the unit of randomization, allocation, duration, and any stratification or variance-reduction approach.
4. Write a pre-registered analysis plan covering the statistical test, peeking policy, multiple comparisons, and how you will handle any mismatch between unit of randomization and unit of analysis.
5. Explain the launch decision rule and identify key experimentation pitfalls such as novelty effects, network interference, SUTVA violations, and sample ratio mismatch.