Context
The Instagram Reels team is testing a new feed ranking tweak that adds a stronger "save-likelihood" feature to the Reels candidate ranker. The PM believes the change could slightly improve top-of-feed engagement by showing more relevant Reels, but wants a rigorous experiment before shipping.
Hypothesis Seed
The proposed change is expected to increase Reels click-through rate (CTR) from the Reels tray into a Reel watch session. Because the model explicitly optimizes for downstream engagement, the team also wants to ensure it does not hurt watch time, IG Save behavior, or user experience in the broader AARRR funnel.
Constraints
- Eligible traffic: 12,000,000 Reels tray impressions per day globally
- Maximum experiment window: 14 days, after which ranking resources will be reallocated
- Randomization must happen in Meta's experimentation platform at the
user_id level
- The team wants 80% power at a 5% two-sided alpha level to detect a 1% relative lift in CTR
- Baseline Reels tray CTR is 20.0%
- False positives are costly because a bad ranker can degrade long-term session quality; false negatives are also costly because ranking launches are limited this quarter
- You may use CUPED with 7-day pre-experiment user-level CTR as a variance reduction covariate
Deliverables
- State the null and alternative hypotheses, and define the primary metric, secondary metrics, and guardrails.
- Calculate the required sample size per arm to detect a 1% relative lift in CTR with 80% power, and translate that into experiment duration given the traffic constraint.
- Choose the unit of randomization, explain the unit of analysis, and describe how you would use CUPED in the analysis.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparisons policy, and SRM checks.
- Give a clear ship / don't-ship recommendation rule that respects guardrails and addresses novelty effect risk on IG Reels.
