Business Context
StreamCart, a grocery delivery app, changed the layout of its weekly promotional email. The marketing team saw a higher click-through rate and wants to know whether the increase is meaningful or just random variation.
Problem Statement
You need to evaluate whether the observed change in click-through rate (CTR) between the old email design and the new design is statistically significant at the 5% level, and whether the effect is large enough to matter for the business.
Given Data
| Group | Emails Delivered | Clicks | CTR |
|---|
| Control (old design) | 52,480 | 3,726 | 7.10% |
| Treatment (new design) | 51,930 | 3,950 | 7.61% |
Additional inputs:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
Requirements
- State the null and alternative hypotheses.
- Compute the sample CTR for each group and the absolute difference in CTR.
- Calculate the pooled proportion and the standard error for a two-proportion z-test.
- Compute the z-statistic and two-tailed p-value.
- Construct a 95% confidence interval for the difference in CTR.
- Decide whether the observed change is statistically significant.
- Explain whether the result is practically meaningful for StreamCart.
Assumptions
- Users were randomly assigned to control and treatment emails.
- Each delivered email corresponds to one independent user outcome.
- Delivery quality is comparable across groups, so CTR differences are not driven by inbox placement.
- The normal approximation is appropriate because both groups have large sample sizes and sufficient clicks/non-clicks.
