Marketing Email A/B Test Significance

Business Context

BrightCart, an online retail marketplace, tested a new promotional email subject line to improve purchase conversion from an email campaign. The marketing team wants to know whether the observed lift is statistically significant or could reasonably be due to random variation.

Problem Statement

Use a two-proportion hypothesis test to determine whether the new subject line changed conversion rate relative to the existing subject line.

Given Data

Additional test settings:

Requirements

1. State the null and alternative hypotheses.
2. Compute the sample conversion rates and the observed lift.
3. Calculate the pooled conversion rate under the null hypothesis.
4. Calculate the standard error for the difference in proportions.
5. Compute the z-statistic and two-tailed p-value.
6. Decide whether the result is statistically significant at the 5% level.
7. Briefly explain what statistical significance means here for the marketing decision.

Assumptions

• Users were randomly assigned to control and treatment.
• Each delivered email corresponds to one independent user outcome.
• Conversion is binary: a user either purchased or did not purchase.
• The normal approximation is appropriate because both groups have large sample sizes and enough conversions/non-conversions.