Business Context
StreamCart tested a new mobile checkout button intended to improve purchase conversion. After a 14-day A/B test, the PM wants to launch because the treatment looks directionally better, but the experiment did not reach statistical significance.
Problem Statement
Assess whether the observed lift is strong enough to justify launch, and quantify what the current test result does and does not tell us.
Given Data
| Group | Users | Purchases | Conversion Rate |
|---|
| Control | 52,400 | 6,026 | 11.50% |
| Treatment | 52,100 | 6,105 | 11.72% |
Additional inputs:
| Parameter | Value |
|---|
| Test duration | 14 days |
| Significance level | 0.05 |
| Desired power | 0.80 |
| PM's minimum practical lift to launch | 0.50 percentage points |
Requirements
- State the null and alternative hypotheses for a two-sided test.
- Run a two-proportion z-test and compute the pooled proportion, standard error, z-statistic, and p-value.
- Compute a 95% confidence interval for the treatment-control difference.
- Determine whether the result is statistically significant at the 5% level.
- Compare the observed lift with the PM's minimum practical threshold.
- Estimate the required sample size per group to detect a 0.50 percentage point absolute lift with 80% power at 5% significance.
- Based on the evidence, explain what you would recommend to the PM: launch, hold, or continue testing.
Assumptions
- Users were randomly assigned and counted once.
- No sample ratio mismatch or major instrumentation issues.
- Conversion is a Bernoulli outcome and normal approximation is valid.
- The primary metric was pre-specified before the test started.
