Context
The Facebook Groups team wants to test a new feature that highlights when a friend recently commented in a group thread, with the goal of increasing meaningful engagement. Because interactions in FB Groups propagate across members, this is a classic network-effects experiment rather than a standard user-level A/B test.
Hypothesis Seed
The team believes that showing a social-context module in group feed and notifications will increase group-thread engagement by making conversations feel more relevant. However, if treated members cause untreated members to re-engage, naive user-level randomization will create interference and bias the estimate.
Constraints
- Eligible population: 120,000 active FB Groups, covering about 24M weekly active group members
- Average eligible traffic: 3.6M member-group visits per day
- Decision deadline: 21 days total, including a 2-day ramp
- False positive cost is high: shipping a noisy feature platform-wide could increase notification fatigue and reduce long-term retention
- False negative cost is moderate: delaying launch by one sprint is acceptable
- The team also wants to monitor AARRR-style engagement movement, especially activation and retention within Groups
Deliverables
- Define the experiment hypothesis, primary metric, secondary metrics, and guardrails, including an explicit MDE.
- Choose the unit of randomization and explain how you will handle network interference, SUTVA concerns, and possible spillovers across group members.
- Calculate the required sample size with real numbers, and translate it into a feasible test duration under the traffic constraints. Include how CUPED could reduce variance using pre-experiment behavior.
- Pre-register the analysis plan: statistical test, peeking policy, SRM checks, multiple-comparison policy, and how you will assess novelty effect / primacy effect over time.
- State a clear ship / don’t-ship / iterate rule that respects guardrails even if the primary metric is statistically significant.
