Business Context

The TurboTax growth team tested a new onboarding prompt on the start flow to increase the percentage of visitors who begin a federal return. The experiment showed a positive lift, but the PM wants to know whether the evidence is strong enough to ship, extend, or discard the change.

Problem Statement

Use the experiment results below to determine whether the observed lift in conversion is statistically significant. Then explain what you would do next if the lift is positive but not statistically significant.

Given Data

Additional test settings:

Requirements

1. State the null and alternative hypotheses.
2. Compute the pooled conversion rate.
3. Calculate the standard error and z-statistic for a two-proportion z-test.
4. Compute the two-sided p-value.
5. Construct a 95% confidence interval for the treatment-control difference.
6. Decide whether the result is statistically significant at the 5% level.
7. Explain what you would recommend to the TurboTax team if the lift is positive but not statistically significant.

Assumptions

• Users were randomly assigned at the user level.
• Each user is counted once.
• No major sample ratio mismatch or instrumentation issues occurred.
• The primary metric is binary conversion: started return or not.