Validate A/B Test for Signup Flow | Dataford Interview Questions

Business Context

StreamCart, a subscription video platform, tested a new signup page intended to increase trial starts. Before acting on the result, the team wants to know whether the experiment is valid and whether the observed lift is trustworthy.

Problem Statement

You are given the experiment results and a few diagnostics. Determine whether the A/B test can be interpreted causally, which assumptions are required for validity, and whether the measured lift is statistically significant.

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Trial starts	6,026	5,950
Conversion rate	11.50%	12.50%
Avg. sessions per user during test	1.18	1.31
Users exposed on both web and mobile	3.1%	8.4%

Additional test settings:

Parameter	Value
Planned traffic split	50% / 50%
Test duration	10 days
Significance level	0.05
Primary metric	Trial start rate

Requirements

State the null and alternative hypotheses for the primary metric.
Test whether the conversion lift is statistically significant using a two-proportion z-test.
Compute a 95% confidence interval for the difference in conversion rates.
Check for sample ratio mismatch against the planned 50/50 allocation.
List the key assumptions required for an A/B test to be valid.
Based on the diagnostics, explain whether the result is fully trustworthy and what you would do next.

Assumptions

User-level randomization was intended.
Trial start is measured once per assigned user.
Use a two-sided test.
Ignore finite population corrections.

Given Data

Metric	Control	Treatment
Assigned users	52,400	47,600
Trial starts	6,026	5,950
Conversion rate	11.50%	12.50%
Avg. sessions per user during test	1.18	1.31
Users exposed on both web and mobile	3.1%	8.4%

Additional test settings:

Parameter	Value
Planned traffic split	50% / 50%
Test duration	10 days
Significance level	0.05
Primary metric	Trial start rate

Requirements

State the null and alternative hypotheses for the primary metric.

Test whether the conversion lift is statistically significant using a two-proportion z-test.

Compute a 95% confidence interval for the difference in conversion rates.

Check for sample ratio mismatch against the planned 50/50 allocation.

List the key assumptions required for an A/B test to be valid.

Based on the diagnostics, explain whether the result is fully trustworthy and what you would do next.