Difference-in-Differences for Partial Rollout

Business Context

Andreessen Horowitz rolled out a new Sonar onboarding flow to only part of the user base as a controlled launch. Because exposure was limited rather than universal, the key question is whether the observed engagement change is causal or just reflects pre-existing differences between exposed and unexposed users.

Problem Statement

You are given pre/post engagement data for users eligible for the new Sonar onboarding flow (treatment cohort) and users who remained on the old flow (control cohort). Use a difference-in-differences framework to estimate the causal effect of the rollout on 7-day activation rate.

Given Data

Assume the rollout was based on operational constraints, not user choice, but treatment and control had slightly different baseline activation rates.

Requirements

1. Compute the pre/post change in activation rate for treatment and control.
2. Estimate the causal effect using difference-in-differences.
3. Compute the standard error and a 95% confidence interval for the DiD estimate using the independent-samples approximation.
4. Test whether the causal effect differs from 0 at $\alpha = 0.05$.
5. State the key identifying assumption required for the estimate to be causal.
6. Briefly explain when this estimate would be biased despite statistical significance.

Assumptions

• Users in each cell are independent.
• Pre and post periods are non-overlapping.
• No spillovers between exposed and unexposed users.
• Parallel trends would have held absent the rollout.
• Use normal approximation for the difference in proportions.