Andreessen Horowitz rolled out a new Sonar onboarding flow to only part of the user base as a controlled launch. Because exposure was limited rather than universal, the key question is whether the observed engagement change is causal or just reflects pre-existing differences between exposed and unexposed users.
You are given pre/post engagement data for users eligible for the new Sonar onboarding flow (treatment cohort) and users who remained on the old flow (control cohort). Use a difference-in-differences framework to estimate the causal effect of the rollout on 7-day activation rate.
| Cohort | Pre-period Users | Pre-period Activated | Pre Activation Rate | Post-period Users | Post-period Activated | Post Activation Rate |
|---|---|---|---|---|---|---|
| Treatment (received new Sonar onboarding) | 18,000 | 3,240 | 18.0% | 19,500 | 4,290 | 22.0% |
| Control (old onboarding) | 42,000 | 7,980 | 19.0% | 41,000 | 8,200 | 20.0% |
Assume the rollout was based on operational constraints, not user choice, but treatment and control had slightly different baseline activation rates.