Unify Databricks Platform Team Support

Project Background

Databricks is standardizing how internal platform engineering supports four user groups on a shared Lakehouse platform: data engineering, data science, ML, and application integration teams. Today, these teams use overlapping but inconsistent workflows across Databricks Workflows, Delta Live Tables, Unity Catalog, MLflow, Model Serving, and partner integrations, creating delivery delays and unclear ownership.

You are the DevOps Engineer responsible for driving a 12-week execution plan with a cross-functional team of 11 people: 4 platform engineers, 2 data engineers, 1 MLOps engineer, 1 security engineer, 1 SRE, 1 product manager, and you. The goal is not to redesign the platform from scratch, but to launch a practical operating model, baseline automation, and onboarding path that improves reliability and speed for all four groups before the next quarter starts.

Key Stakeholders

The Engineering Director wants a fast rollout with minimal disruption. The Head of Data Platform wants stronger governance through Unity Catalog and fewer one-off exceptions. Data science leads want self-service access to MLflow and Model Serving without waiting on platform tickets. Application integration teams want stable APIs and SLAs for downstream consumption. Security is concerned that current cluster policies and secret management are inconsistent.

Constraints

• Timeline: 12 weeks, with executive review in Week 4 and launch readiness review in Week 10
• Budget: $180,000 for contractors, enablement, and temporary migration support
• Team capacity: no new full-time headcount; 2 engineers are only available 50% due to production support
• Dependencies: Unity Catalog migration for 3 critical domains, CI/CD standardization for Databricks Asset Bundles, and approval from security for secret scopes and service principals

Complications

1. One data science team is threatening to bypass the standard platform because their model launch is due in 6 weeks.
2. A key data engineering pipeline in Delta Live Tables has a 4-hour daily SLA breach, consuming SRE attention.
3. The application integration team needs a stable contract for downstream access, but the data platform team wants to change table layouts during the same window.

Your Task

1. Build a 12-week execution plan with milestones, owners, and dependencies.
2. Define how you would prioritize support across data engineering, data science, ML, and application integration teams.
3. Recommend launch scope, trade-offs, and what should be deferred.
4. Define success metrics and a rollout/rollback approach.
5. Identify the top risks and how you would manage stakeholder conflict during execution.