Project Background
Databricks plans to launch a new Unity Catalog lineage experience that shows upstream and downstream dependencies for Delta tables, notebooks, and Databricks SQL dashboards. The feature is strategically important for a flagship customer event in 10 weeks because Sales has committed demos to 15 enterprise accounts, but the current implementation has known gaps in lineage completeness and UI performance.
You are the engineering manager for a team of 9 engineers, 1 product manager, 1 designer, and shared support from QA and SRE. The question is not whether to launch, but how to make the trade-off between shipping quickly and shipping high-quality code while protecting customer trust and platform reliability.
Key Stakeholders
The VP of Product wants a visible launch at the event. The Head of Field Engineering wants a stable demoable experience for strategic prospects. The Security and Compliance lead is concerned that incorrect lineage could mislead customers during audits. The Engineering Director wants to avoid creating long-term maintenance debt that slows the Unity Catalog roadmap next quarter.
Constraints
- Launch deadline: 10 weeks, tied to Databricks Data + AI Summit
- Budget: no additional headcount; only $120,000 for contractor QA and performance testing
- Team capacity: 11 dedicated team members, with 2 backend engineers also spending 20% on production support
- Dependencies: metadata ingestion changes from the Unity Catalog platform team due in Week 4; Databricks SQL frontend integration due in Week 6
- Quality baseline: no Sev-1/Sev-2 incidents in launch week; p95 page load under 2.5 seconds for graphs with up to 500 nodes
Complications
- In Week 3, testing shows lineage completeness is only 87% for notebook-to-table relationships, below the 97% target.
- In Week 5, the platform team slips its metadata API by 10 days, putting the full scope at risk.
- A strategic customer has asked for dashboard lineage in the launch build, but that path has the highest defect rate.
Your Task
- Define what should ship at the event versus what should be deferred.
- Create a 10-week execution plan with milestones, owners, and decision points.
- Propose how you would communicate and justify the speed-versus-quality trade-off to stakeholders.
- Define launch success metrics, guardrails, and a rollback plan.
- Identify the top risks and how you would mitigate them.
