Statistical vs Business Impact in Pricing Test

Business Context

StreamCart, a grocery delivery app, tested a new pricing page that highlights annual-plan savings. The experiment ran on a large share of traffic, and the analytics dashboard shows a statistically significant lift in annual-plan conversion.

Problem Statement

You need to determine whether the observed lift is statistically significant and then explain why that result may still have limited business value.

Given Data

Additional business inputs:

Requirements

1. State the null and alternative hypotheses for a two-sided test.
2. Compute the pooled conversion rate and the standard error.
3. Calculate the z-statistic and p-value for the difference in proportions.
4. Construct a 95% confidence interval for the absolute lift.
5. Estimate the expected incremental gross profit from the observed lift over these 1,000,000 users.
6. Explain how the result can be statistically significant but still have little business impact.
7. Recommend whether StreamCart should roll out the change based on both statistical and practical significance.

Assumptions

• Random assignment was valid and there was no sample ratio mismatch.
• Each user had one exposure and one conversion opportunity.
• Gross profit per additional annual-plan conversion is constant at $18.
• Ignore longer-term retention effects unless explicitly discussed.