

StreamCart, a subscription video platform, tested a new upsell banner on its annual-plan checkout page. The experiment reached a very large audience, and the PM wants to know how to explain a result that is statistically significant but may not matter much financially.
Use the experiment data below to determine whether the treatment increased average revenue per visitor (ARPV), and whether the effect is large enough to matter for the business.
| Group | Sample Size | Mean ARPV | Standard Deviation |
|---|---|---|---|
| Control | 500000 | 12.40 | 40.00 |
| Treatment | 500000 | 12.48 | 40.00 |
Additional business inputs:
| Metric | Value |
|---|---|
| Significance level | 0.05 |
| Minimum business-relevant lift per visitor | 0.25 |
| Monthly visitors to this page | 8000000 |
| One-time engineering + design rollout cost | 180000 |
{"alpha":0.05,"control_n":500000,"control_sd":40,"treatment_n":500000,"rollout_cost":180000,"treatment_sd":40,"monthly_visitors":8000000,"control_mean_arpv":12.4,"treatment_mean_arpv":12.48,"minimum_business_lift":0.25}Output(none)