Business Context
BrightCart, a retail app, tested a new promotional banner to increase email sign-ups. The VP of Marketing saw a p-value in the experiment readout and asked, "What does this actually mean in plain English?"
Problem Statement
You need to evaluate the A/B test statistically and then explain the p-value to a non-technical stakeholder without using misleading language. Use the data below to determine whether the banner likely changed sign-up behavior.
Given Data
| Group | Sample Size | Sign-ups | Conversion Rate |
|---|
| Control (old banner) | 8,400 | 1,008 | 12.0% |
| Treatment (new banner) | 8,100 | 1,053 | 13.0% |
Additional test settings:
| Parameter | Value |
|---|
| Significance level | 0.05 |
| Test type | Two-tailed |
Requirements
- State the null and alternative hypotheses.
- Compute the sample proportions for control and treatment.
- Calculate the pooled proportion and standard error for a two-proportion z-test.
- Compute the z-statistic and two-tailed p-value.
- Decide whether the result is statistically significant at α=0.05.
- Write a 2-3 sentence explanation of the p-value for the VP of Marketing in plain English.
- Briefly explain one common incorrect interpretation of a p-value.
Assumptions
- Users were randomly assigned to control and treatment.
- Each observation is independent.
- Sample sizes are large enough for the normal approximation to hold.
- No major instrumentation or tracking issues affected sign-up counts.
