Respond to Guardrail Metric Regression

Project Background

At Duolingo, the Growth team is running a 4-week experiment on a redesigned lesson-start screen intended to increase paid subscription conversion. The experiment is live to 50% of new users in the US, and early readouts show a +3.8% lift in trial starts. However, a key guardrail metric—Day 7 learner retention—has dropped by 2.1%, and support tickets about confusing onboarding have risen 18%.

Team & Resources

You are the program manager coordinating a cross-functional team of 9: 3 engineers, 1 data scientist, 1 designer, 1 product manager, 1 user researcher, 1 QA lead, and 1 marketing manager. The VP of Growth wants to preserve the conversion gain and launch globally within 3 weeks to hit quarterly revenue targets. The Head of Learning is concerned the retention drop signals lower long-term learner value and wants the test paused immediately.

Constraints

• Only $75,000 remains in the quarter for additional research, engineering work, and experiment reruns
• The team has 21 days before the quarterly launch review
• Engineering capacity is capped at 18 person-days because 2 engineers are shared with a separate payments release
• Analytics instrumentation for retention is delayed by 48 hours, so same-day diagnosis is incomplete
• Legal requires 5 business days notice before any global rollout affecting subscription flows

Complications

1. The CFO has already included the projected conversion lift in the quarter's forecast
2. A parallel onboarding experiment is running in Canada, creating possible interaction effects on retention
3. Customer Support believes the ticket increase is concentrated among older users, but the sample is small

Your Task

1. Propose how you would decide whether to pause, continue, ramp down, or modify the experiment
2. Create a 21-day execution plan with decision points, owners, and stakeholder communication
3. Define the success criteria and guardrail thresholds for launch readiness
4. Recommend a mitigation plan if the conversion gain is real but the retention decline persists
5. Outline a rollback and monitoring plan for the first 14 days after any launch