Conversion Lift Test for Signup Feature

Business Context

StreamCart, a subscription video platform, launched a new signup page feature that shows personalized plan recommendations. The product team wants to know whether the feature improves paid conversion from visitor to subscriber.

Problem Statement

You are asked to design and analyze an experiment to determine whether the new feature increases conversion rate relative to the current signup flow.

Given Data

A 14-day user-level A/B test was run with random assignment at the first eligible visit.

Additional parameters:

Requirements

1. State the null and alternative hypotheses for this experiment.
2. Identify the correct unit of randomization and primary metric.
3. Compute the observed lift in conversion rate.
4. Run a two-proportion z-test and calculate the p-value.
5. Construct a 95% confidence interval for the treatment-control difference.
6. Determine whether the result is statistically significant at $\alpha = 0.05$.
7. Check whether the observed sample size is roughly adequate for the stated minimum detectable effect.
8. Briefly list two guardrail metrics or experimental risks you would monitor before rollout.

Assumptions

• Randomization was implemented correctly and independently across users.
• Each user appears once in the analysis dataset.
• No major tracking bugs or sample ratio mismatch occurred.
• The experiment duration is long enough to cover day-of-week effects.