Email Subject Line Significance Test | Dataford Interview Questions

Your question is Email Subject Line Significance Test. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Business Context

BrightCart, an online retailer, tested a new promotional email subject line to improve open rates. The marketing team wants to know whether the observed lift is statistically significant or could plausibly be due to random variation.

Problem Statement

You are given results from a randomized A/B test comparing the current subject line (control) to a new subject line (treatment). Determine whether the treatment meaningfully changed the email open rate, explain what statistical significance means in this setting, and describe how the p-value relates to the decision.

Given Data

Group	Emails Sent	Opens	Open Rate
Control	8,400	1,764	21.0%
Treatment	8,100	1,863	23.0%

Additional test setting:

Parameter	Value
Significance level (α)	0.05
Test type	Two-sided

Requirements

State the null and alternative hypotheses.
Compute the sample open rates and the observed lift.
Calculate the pooled proportion and standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Decide whether the result is statistically significant at the 5% level.
Explain, in plain language, what the p-value means and what it does not mean.
Compute a 95% confidence interval for the difference in open rates.
Give a business recommendation on whether BrightCart should adopt the new subject line.

Assumptions

Users were randomly assigned to one subject line.
Each email outcome is independent at the user level.
Open tracking is measured consistently across both groups.
The normal approximation for proportions is appropriate because both groups have large sample sizes.