Business Context
PulseFit, a subscription fitness app, tested a new onboarding flow intended to increase paid signup conversion. The product manager wants to know how to interpret the confidence interval from the experiment, not just whether the result is statistically significant.
Problem Statement
You are given the results of a 10-day A/B test comparing the current onboarding flow (control) with a redesigned flow (treatment). Estimate the 95% confidence interval for the difference in conversion rates and explain what that interval means in the context of the product decision.
Given Data
| Group | Sample Size | Paid Signups | Conversion Rate |
|---|
| Control | 18,500 | 2,220 | 12.00% |
| Treatment | 18,900 | 2,419 | 12.80% |
Additional parameters:
| Parameter | Value |
|---|
| Confidence level | 95% |
| Significance level | 0.05 |
| Minimum practically meaningful lift | 0.5 percentage points |
Requirements
- Compute the observed difference in conversion rates between treatment and control.
- Calculate the standard error for the difference in two independent proportions.
- Construct the 95% confidence interval for the treatment effect.
- Interpret the confidence interval in plain English for a product manager.
- State whether the result appears statistically significant and whether it is practically meaningful.
- Explain whether PulseFit should roll out the new onboarding flow based on this evidence alone.
Assumptions
- Users were randomly assigned to control and treatment.
- Each observation is independent and each user appears once.
- The normal approximation for proportions is appropriate because both groups have large sample sizes.
- No major instrumentation or tracking issues affected signup measurement.
