A/B Test for Onboarding Feature

Business Context

PulseFit, a mobile fitness app, launched a new guided onboarding flow intended to increase the percentage of new users who complete their first workout within 7 days. The product manager wants to know whether the feature should be rolled out to all new users.

Problem Statement

You are given results from a 14-day randomized A/B test. Determine whether the new onboarding flow produced a statistically significant improvement in 7-day workout-start conversion, and assess whether the observed lift is large enough to matter for the business.

Given Data

Additional test settings:

Requirements

1. State the null and alternative hypotheses.
2. Compute the observed conversion rates and absolute lift.
3. Run a two-proportion z-test using the pooled standard error.
4. Calculate the two-sided p-value.
5. Construct a 95% confidence interval for the difference in conversion rates.
6. Conclude whether the result is statistically significant and whether you would recommend rollout.
7. Briefly describe key A/B testing checks you would perform before trusting the result.

Assumptions

• Users were randomly assigned at the user level.
• Each user appears once in the analysis.
• No major instrumentation issues affected event logging.
• The metric is binary and sample sizes are large enough for normal approximation.