Business Context
StreamCart tested a new promotional email subject line to improve purchase conversion from an email campaign. The analytics team wants to know not just whether the result is significant, but how to correctly interpret the p-value from the experiment.
Problem Statement
You are given results from a randomized A/B test comparing the current subject line (control) with a new subject line (treatment). Use a two-proportion z-test to evaluate the result, then explain what the p-value means in this experiment.
Given Data
| Group | Sample Size | Purchases | Conversion Rate |
|---|
| Control | 18,500 | 1,480 | 8.00% |
| Treatment | 18,200 | 1,565 | 8.60% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
Requirements
- State the null and alternative hypotheses.
- Compute the sample conversion rates and the observed lift.
- Calculate the pooled proportion and standard error.
- Compute the z-statistic and two-tailed p-value.
- Decide whether the result is statistically significant at α=0.05.
- Interpret the p-value correctly in plain English.
- Briefly explain one common misinterpretation of a p-value in A/B testing.
Assumptions
- Users were randomly assigned to control and treatment.
- Each user received only one version of the email.
- Purchase outcomes are independent across users.
- Sample sizes are large enough for the normal approximation to be reasonable.
