Business Context
FinPilot, a personal finance app, launched a new in-app onboarding checklist intended to improve first-week activation. The product team ran a 14-day user-level A/B test and wants to know whether the feature should be rolled out broadly.
Problem Statement
Evaluate whether the new onboarding checklist increased activation rate relative to the current experience, and assess whether the observed lift is large enough to matter operationally.
Given Data
| Group | Users | Activated Users | Activation Rate |
|---|
| Control | 24,800 | 4,712 | 19.0% |
| Treatment | 25,150 | 5,156 | 20.5% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Desired power | 0.80 |
| Minimum detectable effect to evaluate | 1.0 percentage point |
Requirements
- State the null and alternative hypotheses for the activation-rate comparison.
- Compute the sample proportions, pooled proportion, and standard error for a two-proportion z-test.
- Calculate the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the treatment-minus-control lift.
- Decide whether the result is statistically significant at the 5% level.
- Compare the observed lift with the 1.0 percentage point practical threshold and comment on rollout.
- Briefly explain how you would think about power and whether this test appears adequately sized.
Assumptions
- Random assignment was done at the user level.
- Each user appears once in the experiment dataset.
- No major instrumentation issues or sample-ratio mismatch were detected.
- Activation is measured consistently within 7 days for both groups.
