You manage an engineering team responsible for a customer-facing streaming and account-management platform. Leadership says velocity looks better because story points completed rose from 180 to 235 over the last two quarters, but customer-reported defects also increased from 42 to 61 per month and average cycle time only improved from 14 days to 12 days. During the same period, the team introduced a new on-call rotation, expanded automated test coverage from 48% to 67%, and took ownership of an additional service. You need a clear way to determine whether the team is actually improving over time rather than just shipping more work.
How would you define and measure team improvement in this situation, and how would you separate real improvement from changes caused by scope mix, ownership changes, or reporting artifacts?