Business Context
Splice wants to improve activation for first-time users by replacing its current onboarding with a shorter guided flow. The product team ran a randomized A/B test on new signups and wants to know whether the new flow should be rolled out.
Problem Statement
Use the experiment data below to determine whether the new onboarding flow increases 7-day activation rate for first-time users. Activation is defined as creating a project and exporting at least one audio clip within 7 days of signup.
Given Data
| Group | Users | Activated within 7 days | Activation Rate |
|---|
| Control (old onboarding) | 18,420 | 4,421 | 24.00% |
| Treatment (new onboarding) | 18,615 | 4,748 | 25.51% |
Additional experiment settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Desired power | 0.80 |
| Baseline activation rate for planning | 0.240 |
| Minimum detectable effect for planning | 0.012 |
Requirements
- State the null and alternative hypotheses.
- Compute the observed activation rates and the absolute lift.
- Run a two-proportion z-test using the pooled standard error.
- Compute the two-sided p-value.
- Construct a 95% confidence interval for the treatment-control difference.
- Decide whether the result is statistically significant at the 5% level.
- Assess whether the current sample size was adequate for detecting a 1.2 percentage point lift with 80% power.
- Give a product recommendation and list key experimental design considerations for this onboarding test.
Assumptions
- Randomization occurred at the user level on first signup.
- Each user appears once and outcomes are independent across users.
- No major instrumentation bugs or sample-ratio mismatch occurred.
- The 7-day window is fully observed for all included users.
