Interview Guides

A/B Test for Email Campaign Lift | Dataford Interview Questions - Dataford - Ace your Interview

A/B Test for Email Campaign Lift

Easy

Statistics & Probability

Asked at 17 companies17A/B TestingStatistical SignificanceExperimentation

Also asked at

Problem

Business Context

BrightCart, an online retail marketplace, tested a new promotional email creative against the current version to improve purchase conversion from a campaign send.

Problem Statement

You need to determine whether the new email creative produced a statistically significant lift in conversion rate and whether the observed lift is large enough to justify rollout.

Given Data

Group	Emails Delivered	Purchases	Conversion Rate
Control (current creative)	24,800	1,364	5.50%
Treatment (new creative)	25,200	1,537	6.10%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Confidence level	95%

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample conversion rates and the observed lift in percentage points.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in conversion rates.
Decide whether BrightCart should roll out the new email creative based on both statistical and practical significance.

Assumptions

Users were randomly assigned to control and treatment.
Each delivered email corresponds to one independent user opportunity.
Conversion is binary: a user either purchased or did not purchase.
The normal approximation is appropriate because both groups have large sample sizes and sufficient numbers of conversions and non-conversions.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25200,"control_rate":0.055,"treatment_rate":0.061,"confidence_level":0.95,"control_conversions":1364,"treatment_conversions":1537}

Output(none)

Problem

Business Context

BrightCart, an online retail marketplace, tested a new promotional email creative against the current version to improve purchase conversion from a campaign send.

Problem Statement

You need to determine whether the new email creative produced a statistically significant lift in conversion rate and whether the observed lift is large enough to justify rollout.

Given Data

Group	Emails Delivered	Purchases	Conversion Rate
Control (current creative)	24,800	1,364	5.50%
Treatment (new creative)	25,200	1,537	6.10%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Confidence level	95%

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample conversion rates and the observed lift in percentage points.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in conversion rates.
Decide whether BrightCart should roll out the new email creative based on both statistical and practical significance.

Assumptions

Users were randomly assigned to control and treatment.
Each delivered email corresponds to one independent user opportunity.
Conversion is binary: a user either purchased or did not purchase.
The normal approximation is appropriate because both groups have large sample sizes and sufficient numbers of conversions and non-conversions.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25200,"control_rate":0.055,"treatment_rate":0.061,"confidence_level":0.95,"control_conversions":1364,"treatment_conversions":1537}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Email Campaign LiftEasy

Marketing Email A/B Test SignificanceEasy

Campaign Lift Significance for Email OfferEasy

Next question

A/B Test for Email Campaign Lift

Easy

Statistics & Probability

Asked at 17 companies17A/B TestingStatistical SignificanceExperimentation

Also asked at

Problem

Business Context

BrightCart, an online retail marketplace, tested a new promotional email creative against the current version to improve purchase conversion from a campaign send.

Problem Statement

You need to determine whether the new email creative produced a statistically significant lift in conversion rate and whether the observed lift is large enough to justify rollout.

Given Data

Group	Emails Delivered	Purchases	Conversion Rate
Control (current creative)	24,800	1,364	5.50%
Treatment (new creative)	25,200	1,537	6.10%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Confidence level	95%

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample conversion rates and the observed lift in percentage points.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in conversion rates.
Decide whether BrightCart should roll out the new email creative based on both statistical and practical significance.

Assumptions

Users were randomly assigned to control and treatment.
Each delivered email corresponds to one independent user opportunity.
Conversion is binary: a user either purchased or did not purchase.
The normal approximation is appropriate because both groups have large sample sizes and sufficient numbers of conversions and non-conversions.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25200,"control_rate":0.055,"treatment_rate":0.061,"confidence_level":0.95,"control_conversions":1364,"treatment_conversions":1537}

Output(none)

Problem

Business Context

BrightCart, an online retail marketplace, tested a new promotional email creative against the current version to improve purchase conversion from a campaign send.

Problem Statement

You need to determine whether the new email creative produced a statistically significant lift in conversion rate and whether the observed lift is large enough to justify rollout.

Given Data

Group	Emails Delivered	Purchases	Conversion Rate
Control (current creative)	24,800	1,364	5.50%
Treatment (new creative)	25,200	1,537	6.10%

Additional test settings:

Parameter	Value
Significance level	0.05
Test type	Two-sided
Confidence level	95%

Requirements

State the null and alternative hypotheses for the A/B test.
Compute the sample conversion rates and the observed lift in percentage points.
Calculate the pooled proportion and the standard error for a two-proportion z-test.
Compute the z-statistic and two-sided p-value.
Construct a 95% confidence interval for the difference in conversion rates.
Decide whether BrightCart should roll out the new email creative based on both statistical and practical significance.

Assumptions

Users were randomly assigned to control and treatment.
Each delivered email corresponds to one independent user opportunity.
Conversion is binary: a user either purchased or did not purchase.
The normal approximation is appropriate because both groups have large sample sizes and sufficient numbers of conversions and non-conversions.

Sample Data

Example 1

Input

{"alpha":0.05,"control_n":24800,"treatment_n":25200,"control_rate":0.055,"treatment_rate":0.061,"confidence_level":0.95,"control_conversions":1364,"treatment_conversions":1537}

Output(none)

Your answer

Try one AI text evaluation on us

Get structured feedback, scored against a 4-axis rubric. Premium unlocks unlimited.

0 wordstarget ~200

Up next

A/B Test for Email Campaign LiftEasy

Marketing Email A/B Test SignificanceEasy

Campaign Lift Significance for Email OfferEasy

Next question