Business Context
PulseMail, a marketing automation platform, wants to know whether a new subject line meaningfully improves email click-through rate (CTR). The analytics team ran a randomized A/B test on a campaign sent to similar customer segments.
Problem Statement
Use an appropriate statistical method to determine whether the new subject line produced a statistically significant change in CTR versus the original subject line.
Given Data
| Group | Emails Delivered | Clicks | Click-Through Rate |
|---|
| Control (old subject line) | 18,400 | 1,472 | 8.0% |
| Treatment (new subject line) | 17,900 | 1,557 | 8.7% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
Requirements
- State the null and alternative hypotheses.
- Identify the statistical method you would use and why it is appropriate.
- Calculate the sample proportions for both groups.
- Compute the pooled proportion and standard error.
- Calculate the test statistic and two-tailed p-value.
- Construct a 95% confidence interval for the difference in CTR.
- Conclude whether PulseMail should treat the result as statistically significant.
Assumptions
- Users were randomly assigned to control and treatment.
- Each delivered email corresponds to one independent observation.
- The normal approximation is valid because both groups have sufficiently large sample sizes and enough clicks/non-clicks.
- No major deliverability issues or targeting changes occurred during the experiment.
