Significance Test for Signup Conversion Change

Business Context

StreamCart changed its mobile signup landing page and wants to know whether the observed conversion lift is statistically meaningful or just random variation.

Problem Statement

You are given pre-change and post-change user-level conversion results from a 10-day measurement window. Treat this as a comparison of two independent proportions and determine whether the metric change is statistically significant.

Given Data

Additional parameters:

Requirements

1. State the null and alternative hypotheses.
2. Compute the sample conversion rates and the observed absolute lift.
3. Calculate the pooled proportion and the standard error for a two-proportion z-test.
4. Compute the z-statistic and two-sided p-value.
5. Construct a 95% confidence interval for the difference in conversion rates.
6. Decide whether the change is statistically meaningful or likely noise.
7. Briefly explain whether the result is practically meaningful for the business.

Assumptions

• Users in the before and after periods are independent observations.
• Traffic mix is comparable across periods.
• No major concurrent marketing or pricing changes affected signup behavior.
• Sample sizes are large enough for normal approximation to be reasonable.