Business Context
StreamBox launched an autoplay recommendations shelf on episode-complete pages. Product leadership wants to know whether the shelf creates incremental viewing or simply shifts users from existing discovery surfaces.
Problem Statement
A randomized experiment was run for 21 days at the user level. Control users saw the existing product only; treatment users saw the new shelf in addition to all existing surfaces. To test incrementality versus cannibalization, analyze both total viewing conversion and existing-surface viewing conversion.
Given Data
| Metric | Control | Treatment |
|---|
| Users | 48,200 | 47,900 |
| Users with any episode start within 24h | 12,050 | 12,933 |
| Users with episode start from existing surfaces within 24h | 9,158 | 8,862 |
| Users with episode start from new shelf within 24h | 0 | 2,395 |
Additional notes:
- Total viewing conversion rate = users with any episode start / users
- Existing-surface conversion rate = users who started from legacy surfaces / users
- Significance level: α=0.05
Requirements
- State hypotheses for incremental growth using total viewing conversion.
- Test whether treatment increased total viewing conversion using a one-sided two-proportion z-test.
- Test whether treatment reduced existing-surface viewing conversion using a one-sided two-proportion z-test.
- Compute a 95% confidence interval for the lift in total viewing conversion.
- Quantify net incrementality: compare the gain in total starts to the loss from existing surfaces.
- Conclude whether the feature appears incremental, cannibalizing, or mixed.
Assumptions
- User-level randomization was implemented correctly.
- Each user contributes at most once to each binary metric.
- No major interference between users.
- Normal approximation is valid because sample sizes are large in both groups.
