Business Context
StreamCart, a grocery delivery app, tested a new pricing page that highlights annual-plan savings. The experiment ran on a large share of traffic, and the analytics dashboard shows a statistically significant lift in annual-plan conversion.
Problem Statement
You need to determine whether the observed lift is statistically significant and then explain why that result may still have limited business value.
Given Data
| Group | Sample Size | Annual Plan Conversions | Conversion Rate |
|---|
| Control (old pricing page) | 500,000 | 20,000 | 4.00% |
| Treatment (new pricing page) | 500,000 | 20,750 | 4.15% |
Additional business inputs:
| Metric | Value |
|---|
| Absolute lift in conversion rate | 0.15 percentage points |
| Relative lift | 3.75% |
| Incremental gross profit per annual-plan conversion | $18 |
| One-time engineering + design rollout cost | $220,000 |
| Significance level | 0.05 |
Requirements
- State the null and alternative hypotheses for a two-sided test.
- Compute the pooled conversion rate and the standard error.
- Calculate the z-statistic and p-value for the difference in proportions.
- Construct a 95% confidence interval for the absolute lift.
- Estimate the expected incremental gross profit from the observed lift over these 1,000,000 users.
- Explain how the result can be statistically significant but still have little business impact.
- Recommend whether StreamCart should roll out the change based on both statistical and practical significance.
Assumptions
- Random assignment was valid and there was no sample ratio mismatch.
- Each user had one exposure and one conversion opportunity.
- Gross profit per additional annual-plan conversion is constant at $18.
- Ignore longer-term retention effects unless explicitly discussed.
