Business Context

StreamCart, a subscription video platform, tested a new homepage signup CTA. After 10 days, the product manager wants to ship because treatment looks better on the primary metric.

Problem Statement

You need to determine whether the observed lift in signup conversion is statistically significant and whether the experiment itself may be misleading due to common A/B testing pitfalls.

Given Data

Additional experiment facts:

Requirements

1. State the null and alternative hypotheses for signup conversion.
2. Run a two-proportion z-test on the primary metric.
3. Compute a 95% confidence interval for the treatment-control difference.
4. Check for sample ratio mismatch (SRM) using the expected 50/50 split.
5. Based on the numbers above, identify the main pitfalls that could make this A/B test misleading.
6. Give a recommendation on whether StreamCart should roll out immediately.

Assumptions

• User-level randomization was intended.
• Each assigned user is counted once.
• Signup is a binary outcome.
• Use a two-sided test for the primary metric.
• For the SRM check, use a chi-square goodness-of-fit test with expected counts based on a 50/50 split.