Experiment Significance for Pricing Banner

Business Context

StreamCart, a subscription video platform, tested a new pricing banner on its sign-up page. Product leadership wants to know whether the observed lift is both statistically significant and large enough to matter financially.

Problem Statement

Analyze the A/B test on sign-up conversion and decide whether the treatment should be rolled out. You should evaluate both statistical significance and practical significance.

Given Data

Additional business inputs:

Requirements

1. State the null and alternative hypotheses for a two-sided test.
2. Compute the sample conversion rates and the observed lift.
3. Calculate the pooled proportion and standard error for a two-proportion z-test.
4. Compute the z-statistic and p-value.
5. Construct a 95% confidence interval for the treatment-control difference.
6. Decide whether the result is statistically significant at $\alpha = 0.05$.
7. Decide whether the result is practically meaningful using the minimum lift threshold and expected business value.
8. Give a rollout recommendation and explain any caveats.

Assumptions

• Users were randomly assigned and each user appears once.
• No major tracking bugs or sample ratio mismatch occurred.
• The signup event is binary and independent across users.
• A normal approximation is appropriate because both groups have large sample sizes.