Business Context
LinkUp, a social app, tested a new onboarding flow intended to increase user activation. The experiment improved activation, but trust-and-safety teams noticed a possible increase in user reports for spam or abuse.
Problem Statement
Assess whether the treatment should be rolled out given a statistically significant improvement in the primary metric but a possible degradation in a guardrail metric. You should evaluate both activation and spam-report rates, quantify uncertainty, and make a recommendation.
Given Data
The experiment ran for 21 days with user-level randomization.
| Metric | Control | Treatment |
|---|
| Users | 50,000 | 50,000 |
| Activated users within 7 days | 9,000 | 9,900 |
| Activation rate | 18.0% | 19.8% |
| Users who generated at least one spam report within 7 days | 300 | 360 |
| Spam-report rate | 0.60% | 0.72% |
Use a significance level of 0.05 for each metric.
Requirements
- State null and alternative hypotheses for both activation and spam-report rate.
- Run a two-proportion z-test for activation.
- Run a two-proportion z-test for spam reports.
- Compute 95% confidence intervals for the treatment-minus-control difference for both metrics.
- Determine whether the activation lift is statistically significant.
- Determine whether the spam increase is statistically significant.
- Recommend whether LinkUp should fully roll out, hold, or launch with mitigation, and explain why.
Assumptions
- Randomization was valid and there was no sample ratio mismatch.
- Each user appears once and outcomes are independent across users.
- Normal approximation is acceptable because both groups have sufficiently large counts of successes and failures.
- Ignore secondary segmentation unless needed for interpretation.
