Business Context
StreamCart, a subscription video platform, tested a new pricing banner on its sign-up page. Product leadership wants to know whether the observed lift is both statistically significant and large enough to matter financially.
Problem Statement
Analyze the A/B test on sign-up conversion and decide whether the treatment should be rolled out. You should evaluate both statistical significance and practical significance.
Given Data
| Group | Users Exposed | Sign-ups | Conversion Rate |
|---|
| Control (current banner) | 24,800 | 2,728 | 11.00% |
| Treatment (new banner) | 25,100 | 2,912 | 11.60% |
Additional business inputs:
| Metric | Value |
|---|
| Significance level | 0.05 |
| Minimum practically meaningful lift | 0.40 percentage points |
| Expected net value per additional signup | $18 |
Requirements
- State the null and alternative hypotheses for a two-sided test.
- Compute the sample conversion rates and the observed lift.
- Calculate the pooled proportion and standard error for a two-proportion z-test.
- Compute the z-statistic and p-value.
- Construct a 95% confidence interval for the treatment-control difference.
- Decide whether the result is statistically significant at α=0.05.
- Decide whether the result is practically meaningful using the minimum lift threshold and expected business value.
- Give a rollout recommendation and explain any caveats.
Assumptions
- Users were randomly assigned and each user appears once.
- No major tracking bugs or sample ratio mismatch occurred.
- The signup event is binary and independent across users.
- A normal approximation is appropriate because both groups have large sample sizes.
