Context
The Instagram Reels growth team is testing a new Save entry point that makes it easier to save a Reel directly from the player. Leadership wants a launch recommendation within three weeks because the surface competes for UI space with Share and Send.
Hypothesis Seed
The team believes the new Save affordance will increase downstream retention and sharing intent by helping users curate content they want to revisit, improving the AARRR funnel at the engagement and retention stages. Early reads are directionally positive on IG Save rate, but the PM asks how to decide whether to launch if the final result is still positive but not statistically significant.
Constraints
- Eligible traffic: 18M daily Reels viewers globally, but only 20% can be exposed due to UI engineering limits this quarter
- Maximum experiment runtime: 21 days, including a 2-day ramp
- Baseline user-level daily save rate on Reels viewers: 8.0%
- Small negative effects on session quality are costly; a false positive is considered worse than a false negative
- The team can use CUPED with prior 14-day save behavior from Meta’s experimentation stack to reduce variance
- Virality matters: the PM also cares about K-Factor impact through downstream shares, but shipping should not depend on exploratory virality metrics alone
Task
- Define the hypothesis, primary metric, secondary metrics, and guardrails. Be explicit about the unit of randomization and unit of analysis.
- Compute the required sample size for a pre-registered MDE, translate it into runtime given the traffic cap, and explain whether the test is adequately powered.
- Write the analysis plan, including the statistical test, CUPED usage, peeking policy, multiple-comparison treatment, and how you will check for Sample Ratio Mismatch (SRM).
- State a clear ship / don’t-ship / iterate rule for the case where the estimate is directionally positive but not statistically significant.
- Name the main pitfalls for this Meta setting, including novelty effect, network interference / SUTVA concerns, and any Reels-specific interpretation risks.
