Project Background
At Duolingo, the Growth team is running a 4-week experiment on a redesigned lesson-start screen intended to increase paid subscription conversion. The experiment is live to 50% of new users in the US, and early readouts show a +3.8% lift in trial starts. However, a key guardrail metric—Day 7 learner retention—has dropped by 2.1%, and support tickets about confusing onboarding have risen 18%.
Team & Resources
You are the program manager coordinating a cross-functional team of 9: 3 engineers, 1 data scientist, 1 designer, 1 product manager, 1 user researcher, 1 QA lead, and 1 marketing manager. The VP of Growth wants to preserve the conversion gain and launch globally within 3 weeks to hit quarterly revenue targets. The Head of Learning is concerned the retention drop signals lower long-term learner value and wants the test paused immediately.
Constraints
- Only $75,000 remains in the quarter for additional research, engineering work, and experiment reruns
- The team has 21 days before the quarterly launch review
- Engineering capacity is capped at 18 person-days because 2 engineers are shared with a separate payments release
- Analytics instrumentation for retention is delayed by 48 hours, so same-day diagnosis is incomplete
- Legal requires 5 business days notice before any global rollout affecting subscription flows
Complications
- The CFO has already included the projected conversion lift in the quarter's forecast
- A parallel onboarding experiment is running in Canada, creating possible interaction effects on retention
- Customer Support believes the ticket increase is concentrated among older users, but the sample is small
Your Task
- Propose how you would decide whether to pause, continue, ramp down, or modify the experiment
- Create a 21-day execution plan with decision points, owners, and stakeholder communication
- Define the success criteria and guardrail thresholds for launch readiness
- Recommend a mitigation plan if the conversion gain is real but the retention decline persists
- Outline a rollback and monitoring plan for the first 14 days after any launch
