A Fortune 500 retail customer is standardizing its cloud data platform on Databricks and needs to launch a new near-real-time analytics foundation before the holiday season. You are the DevOps Engineer leading execution for the platform rollout, and the core decision is whether to prioritize a simpler batch-first design using Delta Lake + Databricks Workflows or invest upfront in a lower-latency streaming architecture using Auto Loader, Structured Streaming, and Delta Live Tables. The delivery team has 9 people, and leadership wants a recommendation and executable launch plan within 10 weeks.
The customer CTO wants a scalable architecture that will not need to be reworked in 6 months. The Head of Analytics wants dashboards in production quickly for merchandising and inventory teams. The FinOps lead is concerned about runaway Databricks compute costs, while the security team requires Unity Catalog-based governance and full auditability before production approval. Sales has also positioned this as a referenceable launch for Databricks in the retail vertical.
The project has a fixed budget of $420,000, including partner support. The platform must ingest data from 14 source systems, including 3 POS feeds, 2 e-commerce systems, and 1 CDC feed from an operational database. The first production release is due in 10 weeks, with no additional headcount beyond 5 data engineers, 2 platform engineers, 1 security engineer, and you. Governance setup in Unity Catalog must be completed by Week 4, and production readiness review is scheduled for Week 8.