Measure Engineering Business Value

Business Context

Notion's engineering organization has grown from 120 to 190 engineers in the last 12 months. The CTO is being asked by the CEO and CFO to show whether increased engineering investment is creating measurable business value, not just shipping more features.

Metric Scenario

In the last two quarters, engineering shipped 14 major releases, reduced average page load time from 2.8s to 2.1s, and improved API uptime from 99.5% to 99.9%. Over the same period, self-serve conversion rose from 3.6% to 4.1%, 90-day retention increased from 41% to 44%, enterprise expansion revenue grew 11%, and support tickets per 1,000 active users fell from 18 to 12. However, total engineering spend also increased 22%, and leadership wants a clear framework linking engineering output to business outcomes.

Requirements

1. Define the primary metric or metric set you would use to show that engineering is delivering business value.
2. Distinguish between engineering output metrics, product/customer outcome metrics, and business impact metrics.
3. Propose a metric hierarchy that connects engineering improvements to revenue, retention, and cost efficiency.
4. Explain how you would decompose changes in the top-line metric to identify which engineering initiatives created value.
5. Recommend guardrails to avoid optimizing for speed of delivery at the expense of reliability or customer experience.

Data Available

• product_usage_daily: workspace_id, user_id, date, active_flag, sessions, feature_events, seat_count
• release_log: release_id, ship_date, team, feature_area, release_type, affected_surface
• performance_metrics_daily: date, platform, page_load_ms, api_latency_ms, crash_rate, uptime_pct
• billing_and_revenue_monthly: account_id, month, plan_type, mrr, expansion_mrr, churned_flag
• support_tickets: ticket_id, account_id, created_at, issue_type, severity, linked_product_area
• experiment_results: experiment_id, launch_date, metric_name, treatment_lift, confidence_level