Your question is Correlation vs Causation in Growth Campaigns. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
FitPulse, a subscription fitness app, noticed that users who received more push notifications tended to have higher 30-day retention. The growth team wants to know whether notifications caused the retention lift or whether the relationship is only correlational.
You are given observational data split by user engagement segment. At the aggregate level, users who received notifications appear to retain better. Determine whether that alone supports a causal claim, and quantify the difference between the naive aggregate conclusion and the segment-adjusted conclusion.
| Segment | Notification Group Users | Retained in Notification Group | No-Notification Users | Retained in No-Notification Group |
|---|---|---|---|---|
| High engagement | 8,000 | 5,600 | 2,000 | 1,500 |
| Low engagement | 2,000 | 400 | 8,000 | 1,200 |
Additional information:
| Metric | Value |
|---|---|
| Significance level | 0.05 |
| Total notification users | 10,000 |
| Total no-notification users | 10,000 |