Business Context
CartLane, an e-commerce retailer, changed the checkout button from gray to green and ran a randomized A/B test to see whether the new color increased completed purchases.
Problem Statement
Determine whether the new checkout button color caused a statistically significant increase in sales conversion rate, and quantify the size of the effect.
Given Data
The experiment ran for 14 days at the user level. Each user saw only one version.
| Group | Users Exposed | Purchases | Conversion Rate |
|---|
| Control (gray button) | 52,480 | 6,298 | 12.00% |
| Treatment (green button) | 51,930 | 6,547 | 12.61% |
Additional parameters:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates and the absolute lift.
- Use a two-proportion z-test to test for statistical significance.
- Calculate the p-value.
- Construct a 95% confidence interval for the difference in conversion rates.
- Conclude whether the button color change should be rolled out based on both statistical and practical significance.
Assumptions
- Random assignment was implemented correctly.
- Users are independent observations.
- Conversion is binary: each exposed user either purchased or did not purchase.
- No major concurrent site changes affected checkout behavior during the test window.
- Sample sizes are large enough for normal approximation to the binomial distribution.
A
