Problem
Scenario
You work on a subscription-based creator platform and have launched a new onboarding flow intended to increase the share of new users who start a free trial within 7 days. After launch, you discover assignment was intended to be 50/50 by user ID, but treatment ended up over-representing iOS users and higher-intent traffic from paid acquisition because of an implementation bug in one entry surface. The team still wants to know whether the launch had a real causal impact and whether the result is trustworthy enough to ship broadly.
Constraints
- Eligible traffic: 120,000 new users per week
- Maximum time to reach a decision: 3 weeks total, including any rerun
- Baseline 7-day trial-start rate: 18%
- You should be able to detect at least a 5% relative lift on the primary metric
- Paid conversion rate cannot fall by more than 1% relative
Question
How would you analyze this launch given the imperfect randomization, and how would you redesign or rerun the experiment if needed so that the final ship decision is statistically valid and operationally safe?
Practicing as: Product Growth Analyst interview at ChimeHi, I'll play your Chime interviewer for the Product Growth Analyst role. Candidates describe these interviews as mixed and moderately difficult, so expect me to be professional and fair. Take your time with the question above and answer like we're in the room.
You are practicing as a guest. Sign up free to get your answer graded with AI feedback. Your draft stays right here.
