Measure Engineering Team Improvement

Business Context

At Criteo, you manage the engineering team responsible for a core part of the Retail Media platform used by advertisers and retailers to create, serve, and report on campaigns. Leadership says the team has "improved a lot" over the last two quarters after investments in CI/CD, on-call rotations, and code review standards, but there is no agreed metric framework to prove whether the team is actually getting better over time.

Metric Scenario

Over the last 6 months, deployment frequency increased from 18 to 52 production deploys per month, mean lead time for changes fell from 9.5 days to 3.8 days, Sev-1/Sev-2 incidents dropped from 11 to 6 per quarter, and employee engagement survey scores rose from 72 to 79. However, escaped defect rate increased from 1.8 to 2.4 per 1,000 story points, roadmap completion fell from 84% to 76%, and voluntary attrition increased from 4% to 9% annualized. Product leadership argues the team is faster; SRE argues reliability is still fragile; HR is worried the gains may be unsustainable.

Requirements

1. Define a metric framework for "team improvement over time" for this Criteo engineering team, including one primary KPI and supporting metrics.
2. Separate leading vs lagging indicators and explain why both are needed.
3. Show how you would normalize for changes in team size, project mix, and release volume.
4. Propose a decomposition to diagnose whether improvement is coming from speed, quality, reliability, or team health.
5. Identify 3-5 guardrails to prevent optimizing one metric at the expense of others.
6. Explain how often you would review each metric and what trend would count as meaningful improvement.

Data Available

• jira_issues: issue_id, team_id, issue_type, story_points, created_at, in_progress_at, done_at, release_id
• github_prs: pr_id, repo, author_id, opened_at, merged_at, review_cycles, lines_changed
• deployments: deployment_id, service_name, environment, started_at, completed_at, rollback_flag
• incident_log: incident_id, severity, service_name, opened_at, resolved_at, root_cause_category
• bugs_prod: bug_id, severity, detected_at, linked_release_id, affected_client
• employee_pulse: employee_id, month, engagement_score, burnout_risk, intent_to_stay