Rank Change Effect Beyond Window

Company Context

Meta is testing a new ranking model for Instagram Reels that changes how often users see creator content, ads, and recommended videos in the feed. The experiment ran for 14 days in a few large markets and showed a small lift in short-term ad revenue, but leadership wants to know whether the change will still improve revenue after the initial launch period.

Problem

The experiment likely has a novelty effect, users may react strongly at first and then revert, and the 14-day window may miss delayed effects on retention, session depth, and downstream ad inventory. There is also concern about sample ratio mismatch (SRM) in one treatment cell and about whether the observed lift is robust after adjusting for pre-experiment behavior with CUPED.

Task

1. Explain how you would estimate the incremental revenue impact beyond the immediate experiment window for the Reels ranking change.
2. Define the key metrics you would use in the AARRR funnel to separate short-term engagement lift from longer-term revenue effects.
3. Describe how you would use pre-experiment data, CUPED, and holdout/cohort analysis to project post-window impact.
4. Call out how you would check for SRM, novelty effect, and spillovers across Instagram surfaces such as Reels, Stories, and Feed.
5. Recommend whether Meta should launch, extend the experiment, or run a follow-up holdout before making a decision.

Constraints

• Only 14 days of experiment data are available today.
• The ranking change was shipped only on ig reels, not Facebook Feed or fb groups.
• Revenue is measured as ad revenue per user, but the ranking change may also affect retention and creator supply.
• Engineering can support one additional holdout or follow-up experiment, but not a long multi-quarter study before a decision is needed.