The TurboTax growth team tested a new onboarding prompt on the start flow to increase the percentage of visitors who begin a federal return. The experiment showed a positive lift, but the PM wants to know whether the evidence is strong enough to ship, extend, or discard the change.
Use the experiment results below to determine whether the observed lift in conversion is statistically significant. Then explain what you would do next if the lift is positive but not statistically significant.
| Group | Sample Size | Conversions | Conversion Rate |
|---|---|---|---|
| Control (current onboarding) | 18,400 | 2,208 | 12.00% |
| Treatment (new prompt) | 18,100 | 2,262 | 12.50% |
Additional test settings:
| Parameter | Value |
|---|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Observed absolute lift | 0.50 percentage points |
| Relative lift | 4.17% |