Holdout Decision for Rippling Email

Business Context

Rippling Growth wants to launch a new lifecycle email journey in Rippling HR Cloud to drive admins to complete payroll setup. The team can either run a standard A/B test between two email variants, or reserve a true holdout group that receives no email so they can estimate incremental lift over the existing baseline.

Problem Statement

Decide whether a holdout group is justified for this experiment, using the observed pilot data and the statistical trade-off between measurement value and lost short-term conversions.

Given Data

A 2-week pilot randomly split eligible admins into three groups:

Additional business inputs:

Requirements

1. Test whether the current email is incremental versus no email using a two-proportion z-test.
2. Test whether the new email outperforms the current email using a two-proportion z-test.
3. Compute a 95% confidence interval for the incremental lift of current email versus holdout.
4. Estimate the expected number and dollar value of incremental completions from sending the current email to the full quarterly audience.
5. Based on statistical and business evidence, argue whether future growth experiments on this surface should include a holdout group.
6. Briefly discuss when a holdout would not be worth the cost.

Assumptions

• Random assignment is valid and there is no interference across admins.
• Each admin is counted once.
• The 14-day completion window is the decision metric.
• Ignore operational email costs; focus on opportunity cost from withholding email.