Problem
Scenario
You work on a consumer fintech product and your team has built a new growth feature that could affect both directly exposed users and users they interact with. The team is debating whether to launch it to everyone except for a small holdout group, or to run a standard A/B test, because the feature may have persistent and spillover effects.
Question
How would you decide whether to use a holdout group for this growth feature? Walk through what would make a holdout necessary, what metrics and tradeoffs you would evaluate, and how you would design the experiment so the result is credible.
What Makes Holdout Relevant
- Effects may persist after first exposure
- Untreated users may be affected by treated users
- The business may want a long-term incrementality read
- A standard concurrent control may become contaminated
Practicing as: Product Growth Analyst interview at ChimeHi, I'll play your Chime interviewer for the Product Growth Analyst role. Candidates describe these interviews as mixed and moderately difficult, so expect me to be professional and fair. Take your time with the question above and answer like we're in the room.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
