Business Context
PulseMail, a B2B marketing platform, tested a new subject-line generator on a limited share of outbound campaign traffic. Early results look better than the current template, but the product manager is concerned that the sample is too small to support a rollout decision.
Problem Statement
Use the observed click-through data to assess whether the new subject-line generator truly improves CTR, or whether the apparent lift could plausibly be due to random variation from a small sample.
Given Data
| Group | Emails Delivered | Clicks | Observed CTR |
|---|
| Control (current template) | 420 | 46 | 10.95% |
| Treatment (new generator) | 180 | 29 | 16.11% |
Additional parameters:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-sided |
| Confidence level | 95% |
Requirements
- State the null and alternative hypotheses for the CTR difference.
- Compute the sample CTR in each group and the observed lift.
- Run a two-proportion z-test and calculate the p-value.
- Construct a 95% confidence interval for the difference in CTR.
- Explain whether the result is statistically significant.
- Given the small sample, recommend what you would do next before a full rollout.
Assumptions
- Email recipients were randomly assigned to control and treatment.
- Each delivered email is an independent Bernoulli trial.
- No major deliverability issues or audience-mix shifts occurred during the test.
- The normal approximation is acceptable but should be interpreted cautiously because the treatment sample is relatively small.
