Business Context
StreamCart, a retail marketplace, built a new product-ranking model intended to improve click-through on recommendation widgets. To validate model value, the team ran a user-level A/B test comparing the current ranking model (control) against the new model (treatment).
Problem Statement
Determine whether the new ranking model creates a statistically significant improvement in recommendation click-through rate (CTR), and quantify the likely size of the lift.
Given Data
| Group | Users Exposed | Users Who Clicked | Observed CTR |
|---|
| Control (current model) | 52,400 | 6,131 | 11.7004% |
| Treatment (new model) | 51,900 | 6,492 | 12.5087% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Randomization unit | User |
Requirements
- State the null and alternative hypotheses for the A/B test.
- Compute the sample CTR in each group and the observed absolute lift.
- Calculate the pooled proportion and the standard error for a two-proportion z-test.
- Compute the z-statistic and two-sided p-value.
- Construct a 95% confidence interval for the difference in CTR.
- Conclude whether the model shows statistically significant value and whether the lift is practically meaningful.
Assumptions
- Users were randomly assigned and appear in only one group.
- Click outcome is binary at the user level.
- Sample sizes are large enough for normal approximation.
- No major instrumentation changes occurred during the experiment.
