Tiny Revenue Lift Significance Test

Business Context

StreamCart, a subscription video platform, tested a new upsell banner on its annual-plan checkout page. The experiment reached a very large audience, and the PM wants to know how to explain a result that is statistically significant but may not matter much financially.

Problem Statement

Use the experiment data below to determine whether the treatment increased average revenue per visitor (ARPV), and whether the effect is large enough to matter for the business.

Given Data

Additional business inputs:

Requirements

1. State the null and alternative hypotheses for the difference in mean ARPV.
2. Compute the standard error for the difference in means.
3. Calculate the z-statistic and p-value.
4. Construct a 95% confidence interval for the treatment effect.
5. Compare statistical significance with practical significance using the business threshold of $0.25 per visitor.
6. Estimate the expected monthly incremental revenue if the observed lift holds.
7. Give a recommendation on whether StreamCart should roll out the banner broadly.

Assumptions

• Random assignment was valid and independent across users.
• ARPV is measured over the same attribution window in both groups.
• With this sample size, the sampling distribution of the mean difference is approximately normal by the Central Limit Theorem.
• Ignore downstream retention effects unless stated otherwise.