Design an Email Experiment | Dataford Interview Questions

Your question is Design an Email Experiment. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Business Context

StreamCart, a grocery delivery app, wants to test whether a new promotional email increases 7-day purchase conversion. The marketing team needs an experiment design and a quick read on whether the pilot result justifies a broader rollout.

Problem Statement

You are asked to design the experiment and evaluate the pilot A/B test result. Assume the primary metric is whether a user makes at least one purchase within 7 days of receiving the email.

Given Data

Group	Users Assigned	Purchasers in 7 Days	Conversion Rate
Control (current email)	8,000	640	8.0%
Treatment (new email)	8,200	738	9.0%

Additional design inputs:

Parameter	Value
Baseline conversion rate	8.0%
Minimum detectable effect	1.0 percentage point
Significance level	0.05
Desired power	0.80
Test type	Two-sided

Requirements

Define the null and alternative hypotheses for the A/B test.
Explain a sound experiment design: unit of randomization, primary metric, guardrail metrics, and key bias risks.
Compute the observed treatment effect in percentage points.
Run a two-proportion z-test using the pilot data.
Construct a 95% confidence interval for the difference in conversion rates.
Estimate the required sample size per group for detecting a 1.0 percentage point lift from an 8.0% baseline at 80% power.
State whether the pilot result is statistically significant and whether you would recommend full rollout.

Assumptions

Users were randomly assigned at the user level.
Each user received at most one email during the test window.
Outcomes are independent across users.
No major deliverability issues or targeting changes occurred during the experiment.