Context
The Instagram Reels team is testing a new save-forward ranking feature that surfaces more “saveable” reels higher in feed. Early reads suggest the overall effect on engagement is small, but two key segments appear to react in opposite directions: new Reels viewers improve while heavy Reels viewers decline.
Hypothesis Seed
The team believes emphasizing content with higher predicted IG Save propensity will improve mid-funnel engagement in the AARRR funnel, especially activation/retention behaviors for lighter viewers, without hurting overall Reels consumption or creator ecosystem health. You are asked to design the experiment and explain how to handle a result where segment A is positive and segment B is negative.
Constraints
- Eligible traffic: 18M daily active Instagram users who open Reels at least once/day
- Max experiment window: 14 days; PM wants a decision by the next launch review
- Allocation target: 50/50 after a 1% ramp for instrumentation checks in Meta’s experimentation platform
- Baseline daily user-level IG Save rate on Reels viewers: 12.0%
- The smallest effect worth shipping on the primary metric is +1.5% relative
- False positives are costly because ranking changes can create long-term creator/network effects; false negatives are also costly because Reels growth is a company priority
Task
- Define the null/alternative hypotheses, the primary metric, 2-4 guardrails, and the pre-registered segments you would allow for decision-making.
- Calculate the required sample size and expected runtime, explicitly stating alpha, power, and MDE. Show how you would use CUPED with pre-experiment IG Save behavior.
- Choose the unit of randomization and analysis, and explain how you would analyze opposite segment effects without falling into post-hoc fishing or Simpson’s paradox.
- Write a pre-registered analysis plan covering the main test, segment heterogeneity testing, multiple comparisons, peeking policy, and SRM checks.
- State a clear ship / don’t-ship / iterate rule that respects guardrails, including what you would do if the overall result is neutral but one segment wins and another loses.
