Context
Chime is considering a change to the SpotMe eligibility and limit-ranking service that reprioritizes which members receive higher overdraft coverage in real time. The team believes the new policy will increase approved SpotMe transactions, but it may also create marketplace-style interference because merchant authorization load, fraud models, and shared risk budgets fluctuate over time.
Hypothesis Seed
You need to decide whether this should run as a standard user-level A/B test or a switchback test. The product manager expects the new policy to improve the rate of approved SpotMe-eligible debit card transactions by routing capacity more efficiently during peak periods, but the effect may depend on time-of-day congestion and shared system state.
Constraints
- Eligible traffic: 1.2M SpotMe-eligible debit authorization attempts per day
- Average unique members per day: 420k
- Baseline approval rate: 61.0%
- Maximum experiment window: 21 days
- Engineering can support either member-level randomization or 30-minute switchback windows, not both
- False positives are costly because a bad launch could increase losses and member complaints; false negatives are also costly because SpotMe is a key retention driver
- Risk requires a pre-registered decision rule and no ad hoc mid-test changes
Deliverables
- Decide whether this experiment should be a simple A/B test or a switchback test, and state the assumptions that drive your choice.
- Write the null and alternative hypotheses, define the primary metric, at least three guardrails, and an explicit MDE.
- Calculate the required sample size and translate it into runtime under your proposed design, using the numbers above.
- Specify the experiment design: unit of randomization, allocation, duration, stratification/blocking, and how you will monitor for SRM, interference, and novelty effects.
- Pre-register the analysis plan and give a clear ship / don’t ship / iterate rule that respects guardrails.
Be prepared to explain what evidence would make you reverse your design choice—for example, if you discovered strong time-based spillovers, member-level carryover, or infra constraints that break SUTVA.
