Business Context

Intuit is testing a new referral prompt in Credit Karma that encourages users to invite friends to check personalized offers. Because invited users can affect the behavior of their referrers and vice versa, a standard user-level A/B test violates independence.

Problem Statement

Design and analyze a holdout strategy that accounts for network effects. Assume the growth team randomizes at the referral-cluster level, where a cluster contains one seed user and everyone they directly invited during the experiment window. The primary metric is cluster-level funded-offer conversions per exposed seed.

Given Data

Additional design inputs:

Requirements

1. Explain why user-level randomization is biased here and why cluster holdout is preferable.
2. State the null and alternative hypotheses for the cluster-level treatment effect.
3. Compute the estimated treatment lift, standard error, test statistic, and p-value using a difference-in-means test on cluster averages.
4. Construct a 95% confidence interval for the lift.
5. Compute the design effect implied by the cluster size and intra-cluster correlation.
6. Interpret whether the result is statistically and practically significant, and whether Intuit should use cluster-based holdouts for future networked features.

Assumptions

• Clusters are independent after randomization.
• Cluster means are approximately normal by the central limit theorem.
• Variances may differ across treatment and control, so use an unequal-variance test.