Context
Chime is considering a redesign of the SpotMe enrollment prompt shown in the mobile app home screen and transaction flows. The growth team believes a more prominent, personalized prompt could increase SpotMe enrollment, but some exposure paths cannot be perfectly randomized because the prompt is also triggered by shared merchant and payroll-related states that may spill over across users in the same employer or household.
Hypothesis Seed
The proposed treatment is a new SpotMe prompt that uses personalized copy and a stronger call-to-action. The team expects it to improve 7-day SpotMe enrollment rate among eligible members, but is concerned that imperfect randomization and interference could bias results if users influence one another or if certain surfaces can only be rolled out by employer/payroll cohort.
Constraints
- Eligible traffic: 180,000 SpotMe-eligible members per week in the Chime app
- Maximum decision window: 21 days
- Engineering can support either member-level randomization on some app surfaces or employer/payroll-cohort randomization across all surfaces, but not both simultaneously
- Baseline 7-day SpotMe enrollment among eligible exposed members: 12.0%
- Small wins below +1.0 percentage point absolute are not worth shipping due to design and maintenance cost
- False positives are costly because an over-aggressive prompt may reduce trust and increase support contacts
Deliverables
- Define the null and alternative hypotheses, the primary metric, 2-4 guardrails, and the MDE.
- Choose an experiment design that handles the fact that perfect member-level randomization is not possible everywhere. Be explicit about unit of randomization, unit of analysis, and whether you would use cluster randomization, switchback, or another design.
- Calculate the required sample size per arm and estimate whether the test can finish within 21 days using the traffic provided.
- Pre-register the analysis plan: statistical test, peeking policy, multiple-comparison policy, and how you will diagnose sample ratio mismatch or interference.
- State a clear ship / don’t ship / iterate rule that respects guardrails, especially if the primary metric improves but trust-related guardrails worsen.
