Confidence Interval for Email CTR Lift | Dataford Interview Questions

Welcome to your interview.

The question is on your right: Confidence Interval for Email CTR Lift. Take a moment with it first.

Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes). Discussion and graded submissions share your five interviewer interactions, so spend them well.

Business Context

BrightCart ran an A/B test on a promotional email subject line. The marketing director does not want only a p-value; they want a confidence interval that can be explained clearly to non-technical stakeholders.

Problem Statement

Use the A/B test results below to estimate the lift in click-through rate (CTR) from the new subject line versus the old one, construct a 95% confidence interval for the difference in proportions, and explain how you would present that interval to executives.

Given Data

Group	Emails Delivered	Clicks	Observed CTR
Control (old subject line)	8,400	756	9.00%
Treatment (new subject line)	8,100	810	10.00%

Assume a two-sided significance level of 0.05.

Requirements

State the parameter of interest and the null and alternative hypotheses.
Compute the observed difference in CTR between treatment and control.
Calculate the standard error for the difference in proportions.
Construct the 95% confidence interval for the true CTR lift.
Determine whether the result is statistically significant at the 5% level.
Write a short stakeholder-friendly interpretation of the confidence interval.
Briefly explain one common misinterpretation of a 95% confidence interval and the correct interpretation.

Assumptions

Users were randomly assigned to subject lines.
Each delivered email corresponds to one independent user outcome.
Sample sizes are large enough for the normal approximation to the binomial distribution.
No major deliverability issues differed across groups.