Measure Engineering Delivery Success

Business Context

NovaCloud is a B2B SaaS company with 14 engineering squads shipping customer-facing features and platform improvements. The CTO says the team "feels busier than ever," but stakeholders disagree on whether delivery performance is actually improving.

Metric Scenario

In the last two quarters, story points completed increased from 4,800 to 5,900 (+23%), but on-time delivery of committed roadmap items fell from 82% to 68%. Median cycle time rose from 4.2 days to 6.1 days, Sev-1/Sev-2 incidents increased from 9 to 15, and 30-day post-release defect rate increased from 1.8 to 2.6 defects per 1,000 deploys. Deployment frequency improved from 210 to 290 production deploys per month, while engineering headcount stayed flat at 96.

The CTO wants a clear framework for measuring and reporting engineering delivery success to the executive team and product leadership. They do not want a single vanity metric; they want a balanced scorecard that captures speed, predictability, quality, and business impact.

Requirements

1. Define the primary metric or scorecard you would use to measure engineering delivery success.
2. Explain which metrics are leading indicators vs lagging indicators.
3. Show how you would decompose the recent performance change to identify the main driver(s).
4. Recommend how to report these metrics weekly to engineering managers and monthly to executives.
5. Identify trade-offs or guardrails so that faster shipping does not reduce reliability or customer outcomes.

Data Available

• jira_issues: issue_id, team_id, issue_type, story_points, status, created_at, started_at, completed_at, committed_sprint_id
• sprint_commitments: sprint_id, team_id, committed_items, completed_items, roadmap_flag
• deployments: deploy_id, team_id, service_id, deployed_at, environment, rollback_flag
• incidents: incident_id, severity, service_id, opened_at, resolved_at, root_cause_category
• defects: defect_id, release_id, created_at, severity, source, customer_reported_flag
• roadmap_items: item_id, owner_team, planned_release_date, actual_release_date, revenue_flag, strategic_priority