Business Context

Databricks recently shipped a platform improvement to the Lakeflow Jobs experience in the Databricks workspace: a redesigned job creation flow that reduces setup steps, adds inline validation, and improves error messaging. Two weeks after launch, engineering leadership wants to know whether the change actually helped users rather than just changing click patterns.

Metric Scenario

Before launch, weekly metrics for Lakeflow Jobs were: 42,000 users who opened the job creation flow, 18,900 users who successfully created a job, median time-to-create of 11.5 minutes, Day-7 repeat job creation rate of 24%, and support ticket rate of 38 tickets per 1,000 creators. In the two weeks after launch, 46,000 users opened the flow, 22,100 created a job, median time-to-create fell to 8.2 minutes, Day-7 repeat job creation rate moved to 25%, and support ticket rate fell to 29 per 1,000 creators. However, the VP notes that traffic also shifted toward larger enterprise accounts and asks whether the improvement truly increased user success.

Requirements

1. Define the primary success metric you would use to evaluate whether the Lakeflow Jobs improvement helped users.
2. Identify 3-5 supporting metrics and guardrails across adoption, efficiency, quality, and retention.
3. Explain how you would decompose the metric movement to separate UX impact from traffic-mix changes.
4. State what user segments you would analyze first and why.
5. Describe what conclusion you could and could not make from the available data.

Data Available

• workspace_events: page views, clicks, validation errors, submit attempts, timestamps, workspace_id, user_id
• jobs_events: job created, run started, run succeeded/failed, task count, compute type
• account_dim: account tier, cloud, region, workspace age, seat count
• user_dim: role, persona, tenure, team
• support_tickets: ticket type, severity, linked workspace, linked feature area