Business Context
BrightCart, an online retail platform, launched a simplified checkout page and wants to know whether the new design truly improved purchase conversion or whether the observed lift could be due to random variation.
Problem Statement
You are given A/B test results from a 10-day experiment. Determine whether the treatment checkout produced a statistically significant change in conversion rate versus the control, and quantify the likely size of the effect.
Given Data
| Group | Sample Size | Conversions | Conversion Rate |
|---|
| Control (old checkout) | 18,400 | 2,208 | 12.0% |
| Treatment (new checkout) | 17,900 | 2,327 | 13.0% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
| Confidence level | 95% |
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates and the observed difference.
- Calculate the pooled proportion and the standard error for a two-proportion z-test.
- Compute the z-statistic and two-tailed p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Decide whether BrightCart should roll out the new checkout based on both statistical and practical significance.
Assumptions
- Users were randomly assigned to control and treatment.
- Each observation is an independent user session.
- The normal approximation is valid because both groups have sufficiently large sample sizes and enough conversions/non-conversions.
- No major instrumentation issues or sample-ratio mismatch occurred during the experiment.
