Business Context
The company runs several internal digital products, including a Supplier Portal and Service Desk. Over the last two quarters, leadership has seen mixed signals: feature throughput is up, but customer-reported issues and missed delivery commitments have also increased.
Metric Scenario
You are the Engineering Manager for a 28-person team (4 squads) supporting a supplier portal. In Q1, the team shipped 96 story points per sprint on average, up from 78 in Q4. However, production incidents rose from 11 to 19, P1/P2 incident resolution time increased from 3.8 hours to 6.1 hours, and the percentage of roadmap commitments delivered on time fell from 84% to 71%. Product leadership wants a clear performance scorecard, while the CTO is concerned that the team may be optimizing for output instead of outcomes, quality, and reliability.
Requirements
- Define the core metrics you would use to evaluate this engineering team, separating leading and lagging indicators.
- Explain which metric or small set of metrics should be the primary KPI(s), and why story points alone are insufficient.
- Decompose the recent performance change to identify whether the main issue is delivery predictability, quality, operational load, or team efficiency.
- Propose a dashboard for weekly and monthly review, including guardrails to prevent gaming.
- Recommend actions you would take if throughput rises while reliability and customer experience worsen.
Data Available
- jira_issues: sprint_id, team_id, issue_type, story_points, status, created_at, resolved_at, planned_flag
- deployments_log: deployment_id, service_name, deployed_at, rollback_flag, change_size
- incident_log: incident_id, severity, service_name, opened_at, resolved_at, root_cause_category
- portal_usage: account_id, daily_active_users, task_success_rate, page_latency_ms
- service_desk_tickets: ticket_id, product_area, created_at, resolution_time_hours, csat_score