Problem
Scenario
You work on an HR software platform and your team has redesigned the employee onboarding checklist shown to new admins during company setup. The team believes the new checklist will increase the rate at which admins complete payroll setup, but the baseline conversion rate is noisy because companies arrive with very different levels of pre-existing setup progress. You want to use CUPED with pre-experiment covariates to improve sensitivity and detect a smaller effect without extending the test.
Constraints
- Eligible traffic: 18,000 new company admins per week
- Maximum experiment duration: 3 weeks
- 50/50 allocation after a 1-day instrumentation ramp
- Support-contact rate cannot increase by more than 0.3 percentage points
- Payroll setup completion cannot be delayed by more than 0.5 days on average
Question
How would you design this experiment so that CUPED is used appropriately to improve power while keeping the analysis valid? Explain how you would choose the covariate, define success and guardrails, size the test, and decide whether to ship.
Practicing as: Data Scientist interview at GoodyearHi, I'll play your Goodyear interviewer for the Data Scientist role. Candidates describe these interviews as mostly positive and moderately difficult, so expect me to be friendly and conversational. Take your time with the question above and answer like we're in the room.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
