Chime tested a new onboarding experience in the Chime app designed to increase early engagement after signup. The experiment improved 7-day engagement, but the product team is concerned because it may slightly reduce initial signup completion.
Decide whether Chime should launch the new onboarding experience. You should evaluate both the primary downside risk on signup completion and the upside on early engagement, then make a recommendation using statistical evidence and practical impact.
The experiment ran for 14 days with user-level randomization.
| Metric | Control | Treatment |
|---|---|---|
| Exposed users | 50,000 | 50,000 |
| Signup completions | 11,000 | 10,650 |
| Signup completion rate | 22.0% | 21.3% |
| Engaged in first 7 days among completed signups | 4,950 | 5,112 |
| 7-day engagement rate among completed signups | 45.0% | 48.0% |
| Significance level |
Assume the business values one completed signup at $42 in expected contribution margin and one additional engaged user at $9 in incremental long-term value.