Randomization for Instagram Reels Test

Context

The Instagram Reels team wants to test a new interaction change: moving the Follow CTA higher on the Reels viewer and adding a subtle animation after 2 seconds of watch time. The product team believes this could increase creator follows and downstream Reels engagement, but it may also distract from viewing or create inconsistent experiences if users see different variants across sessions.

Hypothesis Seed

The proposed UI change will increase the rate at which viewers follow creators from Reels, with minimal harm to watch behavior and ad monetization. Your job is to design the experiment end-to-end, including the correct randomization unit for Instagram Reels.

Constraints

• Eligible traffic: 12M daily active Instagram users who watch at least one Reel per day
• Average eligible users generate 6 Reels viewer sessions/day
• Maximum experiment duration: 14 days
• Allocation can be up to 50/50 after a safe ramp
• Baseline creator-follow-from-Reels rate: 3.0% per eligible user-day
• Product leadership wants to detect at least a 5% relative lift in the primary metric
• False positives are costly because a bad UI could hurt long-term Reels consumption and creator trust; false negatives are acceptable if the effect is very small

Deliverables

1. State the null and alternative hypotheses, and decide whether the test should be one-sided or two-sided.
2. Define the primary metric, 2-4 guardrail metrics, and at least one secondary metric. Explicitly state the unit of analysis and the minimum detectable effect.
3. Choose the unit of randomization for this Instagram Reels test, justify it, and explain any mismatch with the analysis unit.
4. Calculate the required sample size and expected duration using the traffic above.
5. Pre-register an analysis plan, including the statistical test, peeking policy, multiple-comparison handling, and how you would handle pitfalls such as novelty effects, network interference, SUTVA violations, and sample ratio mismatch.