
StreamCart, a grocery delivery app, tested a new pricing page that highlights annual-plan savings. The experiment ran on a large share of traffic, and the analytics dashboard shows a statistically significant lift in annual-plan conversion.
You need to determine whether the observed lift is statistically significant and then explain why that result may still have limited business value.
| Group | Sample Size | Annual Plan Conversions | Conversion Rate |
|---|---|---|---|
| Control (old pricing page) | 500,000 | 20,000 | 4.00% |
| Treatment (new pricing page) | 500,000 | 20,750 | 4.15% |
Additional business inputs:
| Metric | Value |
|---|---|
| Absolute lift in conversion rate | 0.15 percentage points |
| Relative lift | 3.75% |
| Incremental gross profit per annual-plan conversion | $18 |
| One-time engineering + design rollout cost | $220,000 |
| Significance level | 0.05 |
{"alpha":0.05,"control_n":500000,"treatment_n":500000,"control_rate":0.04,"rollout_cost":220000,"absolute_lift":0.0015,"relative_lift":0.0375,"treatment_rate":0.0415,"control_conversions":20000,"treatment_conversions":20750,"incremental_gross_profit_per_conversion":18}Output(none)