Business Context
Motive’s growth team tested a new onboarding flow in the Motive Driver App intended to increase the share of newly invited drivers who complete account setup within 7 days. Product leadership wants to know whether the observed lift is real or just sampling noise.
Problem Statement
Use the experiment results below to determine whether the change in 7-day onboarding completion rate is statistically meaningful.
Given Data
| Group | Sample Size | Completed Onboarding in 7 Days | Completion Rate |
|---|
| Control (current flow) | 18,420 | 8,105 | 44.00% |
| Treatment (new flow) | 18,115 | 8,250 | 45.54% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
Requirements
- State the null and alternative hypotheses.
- Compute the sample proportions and the absolute lift.
- Calculate the pooled proportion and standard error for a two-proportion z-test.
- Compute the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the difference in completion rates.
- Decide whether the metric change is statistically meaningful.
- Briefly explain whether the result is also practically meaningful for Motive.
Assumptions
- Drivers were randomly assigned to control and treatment.
- Each driver appears once in the analysis.
- The metric is binary: completed onboarding within 7 days or not.
- No major instrumentation changes occurred during the experiment.
- The normal approximation is valid because both groups have large sample sizes and enough successes/failures.
