Business Context
StreamCart ran an A/B test on a new mobile checkout layout. The product manager wants a recommendation not only on whether the result is statistically significant, but also on whether to communicate the outcome using a p-value, a confidence interval, or both.
Problem Statement
Analyze the experiment result for conversion rate and decide how to communicate it to two audiences: (1) the product manager deciding on rollout and (2) the executive team reviewing experiment impact.
Given Data
| Group | Sample Size | Conversions | Conversion Rate |
|---|
| Control | 18,400 | 2,208 | 12.0% |
| Treatment | 18,100 | 2,317 | 12.8% |
Additional inputs:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Confidence level | 95% |
The observed lift is 0.8 percentage points.
Requirements
- State the null and alternative hypotheses for a two-sided test.
- Compute the pooled proportion and the standard error for the hypothesis test.
- Calculate the z-statistic and two-sided p-value.
- Compute the 95% confidence interval for the difference in conversion rates.
- Decide whether the result is statistically significant at the 5% level.
- Explain when a p-value is the better communication tool and when a confidence interval is better.
- Recommend what StreamCart should report to the product manager and to executives.
Assumptions
- Users were randomly assigned to control and treatment.
- Each observation is independent.
- Sample sizes are large enough for normal approximation.
- No major instrumentation or logging issues affected conversion counts.
