A/B Testing Variance and Variability | Dataford Interview Questions

Your question is A/B Testing Variance and Variability. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Problem

A/B testing and variance understanding.

Explain how you would turn this into a practical online experiment. Use clearly stated assumptions where numerical inputs are not provided. Address metric definition, variance estimation, sample-size calculation with an explicit MDE, randomization, statistical analysis, and interpretation. Include guardrails and explain how you would avoid misleading conclusions from peeking, sample ratio mismatch, or mismatched analysis and randomization units.

Deliverables:

State the null and alternative hypotheses.
Define primary, secondary, and guardrail metrics.
Calculate illustrative sample size and test duration.
Provide a pre-registered analysis and decision rule.
Identify major sources of variance and experimental risk.

Your question is A/B Testing Variance and Variability. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

A/B testing and variance understanding.

Deliverables:

State the null and alternative hypotheses.
Define primary, secondary, and guardrail metrics.
Calculate illustrative sample size and test duration.
Provide a pre-registered analysis and decision rule.
Identify major sources of variance and experimental risk.