Context
Meta is considering a change to the Facebook Reels share sheet: moving the most-likely recipients to the top and enlarging the first row of share targets. The team wants to know whether this improves sharing without harming downstream engagement quality.
Hypothesis Seed
The product hypothesis is that a more prominent, personalized share sheet reduces friction and increases the probability that a Reel viewer sends a share. In experimentation terms, this means explicitly balancing the risk of a Type I error (shipping a UI that appears to help but does not truly improve sharing) against a Type II error (failing to ship a genuinely beneficial change because the test is underpowered).
Constraints
- Eligible traffic: 1.2M Facebook Reels viewers per day globally
- Only 60% of viewers open the share sheet; randomization must occur before that event to avoid selection bias
- Maximum runtime: 14 days, including one full weekend cycle
- The team will only ship if the primary metric improves materially and key guardrails do not regress
- False positives are costly because a bad share-sheet change can reduce session quality and create spammy sharing; false negatives are also costly because Reels sharing is a strategic growth lever
Deliverables
- Define the null and alternative hypotheses, and explain the Type I vs. Type II error tradeoff in this product context.
- Specify the primary metric, 2-4 guardrails, at least one secondary metric, and an explicit MDE.
- Calculate the required sample size and expected duration using the traffic constraint above.
- Choose the unit of randomization and describe the experiment design, including allocation, stratification, and how you will avoid peeking.
- Pre-register an analysis plan and a ship / don't-ship rule that respects guardrails, and list the main experimentation pitfalls you would monitor.
