Diagnose Misleading Signup A/B Test | Dataford Interview Questions

Business Context

StreamCart, a subscription video platform, tested a new homepage signup CTA. After 10 days, the product manager wants to ship because treatment looks better on the primary metric.

Problem Statement

You need to determine whether the observed lift in signup conversion is statistically significant and whether the experiment itself may be misleading due to common A/B testing pitfalls.

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Signups	6,026	5,950
Observed conversion rate	11.50%	12.50%
Expected traffic split	50%	50%

Additional experiment facts:

Item	Value
Test duration	10 days
Significance level	0.05
Number of dashboard checks during run	8
Other metrics reviewed post hoc	6
Day 1 treatment conversion rate	14.2%
Days 2-10 treatment conversion rate	12.3%

Requirements

State the null and alternative hypotheses for signup conversion.
Run a two-proportion z-test on the primary metric.
Compute a 95% confidence interval for the treatment-control difference.
Check for sample ratio mismatch (SRM) using the expected 50/50 split.
Based on the numbers above, identify the main pitfalls that could make this A/B test misleading.
Give a recommendation on whether StreamCart should roll out immediately.

Assumptions

User-level randomization was intended.
Each assigned user is counted once.
Signup is a binary outcome.
Use a two-sided test for the primary metric.
For the SRM check, use a chi-square goodness-of-fit test with expected counts based on a 50/50 split.

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Signups	6,026	5,950
Observed conversion rate	11.50%	12.50%
Expected traffic split	50%	50%

Additional experiment facts:

Item	Value
Test duration	10 days
Significance level	0.05
Number of dashboard checks during run	8
Other metrics reviewed post hoc	6
Day 1 treatment conversion rate	14.2%
Days 2-10 treatment conversion rate	12.3%

Requirements

State the null and alternative hypotheses for signup conversion.

Run a two-proportion z-test on the primary metric.

Compute a 95% confidence interval for the treatment-control difference.

Check for sample ratio mismatch (SRM) using the expected 50/50 split.

Based on the numbers above, identify the main pitfalls that could make this A/B test misleading.

Give a recommendation on whether StreamCart should roll out immediately.