Context
The Facebook Reels team wants to test a new share-sheet entry point that makes it easier to share a Reel to Messenger or WhatsApp. Before launch, they need to determine whether the experiment can detect a meaningful lift within the available traffic window.
Hypothesis Seed
The proposed UI change places the share action in a more prominent position on the Reel viewer. Product believes this will increase the probability that a Reel viewer shares at least one Reel, but they do not want to ship if the change hurts session quality or creates accidental shares.
Constraints
- Eligible traffic: 1.2M Facebook Reels viewers per day globally
- Maximum experiment duration: 14 days, including ramp
- Randomization must happen at the
user_id level to avoid a user seeing different share-sheet designs across sessions
- Baseline user-level daily share rate: 8.0%
- The team wants 80% power at a 5% two-sided significance level
- Small wins are not enough: the minimum detectable effect should be a 5% relative lift in share rate
- False positives are costly because shipping a noisy UI change to Reels affects a large surface; false negatives are acceptable if the effect is below the MDE
Deliverables
- Define the null and alternative hypotheses, the primary metric, and 2-4 guardrail metrics.
- Compute the required sample size per arm using the stated baseline, alpha, power, and MDE; then translate that into expected runtime given available traffic.
- Specify the experiment design: unit of randomization, allocation, duration, and any stratification or blocking.
- Pre-register an analysis plan covering the statistical test, peeking policy, multiple-comparison handling, and how you will validate experiment integrity.
- State a clear ship / don't-ship / iterate rule that respects both the primary metric and guardrails, and identify key experimentation pitfalls.
