A/B Test for Onboarding Flow

Business Context

Splice wants to improve activation for first-time users by replacing its current onboarding with a shorter guided flow. The product team ran a randomized A/B test on new signups and wants to know whether the new flow should be rolled out.

Problem Statement

Use the experiment data below to determine whether the new onboarding flow increases 7-day activation rate for first-time users. Activation is defined as creating a project and exporting at least one audio clip within 7 days of signup.

Given Data

Additional experiment settings:

Requirements

1. State the null and alternative hypotheses.
2. Compute the observed activation rates and the absolute lift.
3. Run a two-proportion z-test using the pooled standard error.
4. Compute the two-sided p-value.
5. Construct a 95% confidence interval for the treatment-control difference.
6. Decide whether the result is statistically significant at the 5% level.
7. Assess whether the current sample size was adequate for detecting a 1.2 percentage point lift with 80% power.
8. Give a product recommendation and list key experimental design considerations for this onboarding test.

Assumptions

• Randomization occurred at the user level on first signup.
• Each user appears once and outcomes are independent across users.
• No major instrumentation bugs or sample-ratio mismatch occurred.
• The 7-day window is fully observed for all included users.