Significance Test for Signup Conversion

Business Context

StreamCart, a subscription video platform, changed the homepage call-to-action button and wants to know whether the observed change in signup conversion is real or just random variation.

Problem Statement

You are given results from a 10-day A/B test comparing the old homepage (control) with the new homepage (treatment). Determine whether the change in signup conversion rate is statistically significant at the 5% level.

Given Data

Additional test settings:

Requirements

1. State the null and alternative hypotheses.
2. Compute the sample conversion rates for both groups.
3. Calculate the pooled proportion and the standard error for a two-proportion z-test.
4. Compute the z-statistic and two-tailed p-value.
5. Construct a 95% confidence interval for the difference in conversion rates.
6. Decide whether the observed lift is statistically significant or likely noise.
7. Briefly explain what the result means for the rollout decision.

Assumptions

• Users were randomly assigned to control and treatment.
• Each user appears only once in the experiment.
• The normal approximation is valid because both groups have large sample sizes and enough conversions/non-conversions.
• No major instrumentation issues or sample-ratio mismatch occurred during the test.