Business Context

A Meta team is testing a new Instagram Reels ranking tweak intended to increase the IG Save rate, a key downstream engagement signal in the AARRR funnel for retained creators and viewers. The experiment is analyzed in Meta’s standard A/B testing workflow, with CUPED-adjusted estimates available from pre-experiment save behavior.

Problem Statement

Use the experiment setup below to explain Type I and Type II errors in a concrete Meta setting. Then quantify the probabilities of each under the stated assumptions.

Given Data

Assume randomization is valid, the SRM check passed, and the normal approximation is appropriate. Treat the CUPED adjustment as reducing the standard error by a factor of $\sqrt{0.75}$.

Requirements

1. State the null and alternative hypotheses.
2. Give a concrete Type I error example in this Reels experiment.
3. Give a concrete Type II error example in this Reels experiment.
4. Compute the probability of a Type I error at the stated significance level.
5. Compute the Type II error probability when the true lift is +0.30 percentage points and power is 80%.
6. Using the observed lift of +0.05 percentage points, calculate the z-statistic and p-value with CUPED-adjusted standard error.
7. Conclude whether this run would lead to rollout, and explain the business implications of both error types, including novelty effect risk.

Assumptions

• User-level randomization on Instagram Reels viewers
• No Sample Ratio Mismatch concern because SRM p-value is not significant
• CUPED uses stable pre-experiment save behavior
• No major interference across users