Product Context
A customer needs a Databricks-based platform to ingest data from a third-party system, clean and standardize it, and publish business reports on a 10-minute cadence. The users are analysts and operations teams who need fresh, trustworthy metrics without manually stitching together files or APIs.
Scale
- Source volume: about 50 million records per day, arriving in micro-batches and occasional backfills
- Peak ingest rate: roughly 5,000 records per second during source bursts
- Reporting SLA: new data visible in dashboards within 10 minutes of source arrival
- Query load: 200 to 500 concurrent dashboard users during business hours
- Data footprint: around 2 TB/day raw, 300 to 500 GB/day curated after cleaning and deduplication
Task
Design the end-to-end solution on Databricks and explain how you would use the platform components.
- Clarify the functional and non-functional requirements, including freshness, correctness, and availability
- Propose the ingestion and transformation architecture using Databricks services such as Auto Loader, Delta Lake, Structured Streaming, and Delta Live Tables where appropriate
- Explain how you would handle schema drift, late-arriving data, deduplication, and data quality checks
- Describe how the curated data becomes business reports every 10 minutes, including serving and refresh strategy
- Define how you would validate the pipeline offline and monitor it in production
- Identify likely failure modes, such as source outages, bad upstream schema changes, and report staleness, and how you would mitigate them
Constraints
- The third-party source is not under the customer’s control, so schema changes and partial outages are expected
- Reports must be consistent and auditable, not just fast
- The solution should minimize manual operations and avoid brittle point-to-point scripts
- Cost matters, so the design should avoid overprovisioning compute for low-traffic periods
