Facebook Groups Network-Effects Test

Context

The Facebook Groups team wants to test a new feature that highlights when a friend recently commented in a group thread, with the goal of increasing meaningful engagement. Because interactions in FB Groups propagate across members, this is a classic network-effects experiment rather than a standard user-level A/B test.

Hypothesis Seed

The team believes that showing a social-context module in group feed and notifications will increase group-thread engagement by making conversations feel more relevant. However, if treated members cause untreated members to re-engage, naive user-level randomization will create interference and bias the estimate.

Constraints

• Eligible population: 120,000 active FB Groups, covering about 24M weekly active group members
• Average eligible traffic: 3.6M member-group visits per day
• Decision deadline: 21 days total, including a 2-day ramp
• False positive cost is high: shipping a noisy feature platform-wide could increase notification fatigue and reduce long-term retention
• False negative cost is moderate: delaying launch by one sprint is acceptable
• The team also wants to monitor AARRR-style engagement movement, especially activation and retention within Groups

Deliverables

1. Define the experiment hypothesis, primary metric, secondary metrics, and guardrails, including an explicit MDE.
2. Choose the unit of randomization and explain how you will handle network interference, SUTVA concerns, and possible spillovers across group members.
3. Calculate the required sample size with real numbers, and translate it into a feasible test duration under the traffic constraints. Include how CUPED could reduce variance using pre-experiment behavior.
4. Pre-register the analysis plan: statistical test, peeking policy, SRM checks, multiple-comparison policy, and how you will assess novelty effect / primacy effect over time.
5. State a clear ship / don’t-ship / iterate rule that respects guardrails even if the primary metric is statistically significant.