Business Context

PulseApp, a consumer productivity app, noticed that users who receive more push notifications tend to have higher 7-day retention. The product manager wants to know whether sending more notifications actually causes retention to increase, or whether the relationship is driven by more engaged users naturally receiving more notifications.

Problem Statement

You are given both observational data and a follow-up randomized experiment. Use the numbers below to explain why correlation alone is not enough for causal claims, and determine whether the experiment provides evidence of a causal effect.

Given Data

Observational analysis

A simple user-level regression on the observational sample estimated that each additional notification is associated with a 4.0 percentage point increase in probability of 7-day retention.

Randomized experiment

Use a two-sided test with significance level $\alpha = 0.05$.

Requirements

1. Quantify the observational association in retention between high- and low-activity users.
2. Explain why that association does not identify a causal effect of notifications.
3. Test whether the experimental lift is statistically significant using a two-proportion z-test.
4. Compute a 95% confidence interval for the experimental retention lift.
5. State the business conclusion: should PulseApp claim that more notifications cause higher retention?

Assumptions

• Random assignment in the experiment was valid.
• Each user appears once.
• Normal approximation is appropriate for the experiment because both groups are large.