Business Context

You are an Engineering Manager for Autodesk Fusion. Over the last two quarters, leadership has asked whether your team is actually improving, not just shipping more features. Your team owns a collaboration workflow used by design teams to create, review, and share CAD changes.

Metric Scenario

In Q1, the team shipped 14 releases, resolved 320 bugs, and delivered 11 roadmap items. In Q2, the team shipped 18 releases, resolved 410 bugs, and delivered 13 roadmap items. However, customer-reported defects per 1,000 weekly active accounts increased from 2.8 to 3.4, median API latency for save/sync improved from 780 ms to 620 ms, 30-day account retention for collaboration users moved from 71% to 73%, and feature adoption for the new markup workflow rose from 24% to 39% of eligible accounts. Employee pulse survey scores for engineering health improved from 6.4 to 7.1 out of 10, while average cycle time increased from 5.2 to 6.0 days.

Stakeholders want a data-driven answer to: "Is the team improving over time, and how should we measure it going forward?"

Requirements

1. Define a KPI framework that distinguishes output metrics from outcome metrics for this team.
2. Choose 1 primary metric and 3-5 supporting metrics to evaluate whether the team is improving.
3. Explain how you would normalize for release volume, seasonality, and changing user mix.
4. Identify which Q2 changes indicate real improvement versus possible metric traps.
5. Propose a dashboard and review cadence for tracking improvement over the next 6 months.

Data Available

• fusion_usage_events: account_id, user_id, session_id, feature_name, event_name, timestamp, platform
• release_log: release_id, release_date, feature_area, incident_flag, rollback_flag
• jira_issues: issue_id, issue_type, severity, created_at, resolved_at, team_id
• support_cases: case_id, account_id, product_area, severity, created_at, root_cause
• service_metrics: endpoint, p50_latency_ms, p95_latency_ms, error_rate, date
• account_retention: account_id, cohort_month, active_30d, active_90d, plan_tier