FitMeal, a subscription meal-planning app, tested a new onboarding flow intended to increase paid trial starts. After a 21-day A/B test, the treatment looked better than control, but the analytics dashboard flagged the result as not statistically significant.
Evaluate whether the experiment provides enough evidence to ship the new onboarding flow, and explain how you would assess a directionally positive but statistically non-significant result.
| Group | Users | Paid Trial Starts | Conversion Rate |
|---|---|---|---|
| Control | 24,800 | 2,232 | 9.00% |
| Treatment | 24,950 | 2,333 | 9.35% |
Additional inputs:
| Parameter | Value |
|---|---|
| Absolute lift observed | 0.35 percentage points |
| Relative lift observed | 3.89% |
| Significance level | 0.05 |
| Test type | Two-sided |
| Planned minimum detectable effect (absolute) | 0.60 percentage points |