Project Background
You are a Product Growth Analyst supporting the Instagram Reels team at Meta. A PM wants to decide whether to launch a new Reels ranking change that increased short-term engagement in an A/B test, but your analysis shows the story is more nuanced: the treatment improved top-of-funnel AARRR activation and Instagram Save rate, while hurting 7-day retention for some creator-heavy cohorts after adjusting with CUPED. There is also a possible Sample Ratio Mismatch (SRM) in one major market and evidence of a Novelty Effect in the first 72 hours.
The project team includes 1 PM, 2 data scientists, 4 engineers, 1 UXR partner, and you as the analyst. The decision must be made before the next Reels release train in 10 business days because the ranking model freeze is already scheduled.
Key Stakeholders
The Reels PM wants a simple go/no-go recommendation for launch. Engineering wants a decision within 5 days to avoid missing the model freeze. The Growth Director cares about K-Factor lift and creator sharing, while the Integrity partner is concerned that rushing a launch with unresolved SRM could undermine trust in the experiment. These priorities are not fully aligned.
Constraints
- Decision deadline: 10 business days
- Engineering freeze: 5 business days
- No new experiment rerun before freeze
- Only 1 analyst and 1 data scientist available full-time
- Analysis must use existing experiment logs from Scuba and dashboards in Meta's internal experimentation stack
Complications
- The PM is non-technical and has previously over-indexed on headline lifts without understanding CUPED-adjusted results.
- The Growth Director has already referenced the early +3.8% Reels shares lift in a weekly review, before the Novelty Effect and SRM concerns were surfaced.
- Facebook Groups distribution may amplify network effects, making K-Factor interpretation less straightforward.
Your Task
- Create a communication and decision plan for the next 10 business days.
- Explain how you would present the analytical finding to the non-technical PM without losing critical nuance.
- Define the launch recommendation, fallback options, and decision criteria.
- Identify the top execution risks and how you would manage stakeholder alignment.
- Specify the artifacts you would deliver before the model freeze and before final launch review.
