Context
HashedIn by Deloitte supports multiple client-facing data platforms where batch and streaming pipelines are released several times per week. The current process relies on manual promotion of Apache Airflow DAGs, dbt models, and Spark jobs, which has led to broken dependencies, failed backfills, and production data quality regressions.
You are asked to design a low-risk CI/CD process for a data engineering team that needs frequent releases across development, staging, and production while maintaining reliability for analytics and downstream ML consumers.
Scale Requirements
- Pipelines: 180 Airflow DAGs, 65 dbt models, 25 Spark jobs
- Release frequency: 20-30 production deployments per week
- Data volume: 12 TB/day batch + 150K events/sec streaming peak
- Latency SLOs: batch pipelines available by 6:00 AM IST; streaming freshness < 3 minutes
- Environments: dev, QA, staging, prod across AWS
- Recovery target: rollback or forward-fix within 15 minutes
Requirements
- Design a CI/CD workflow for pipeline code, SQL transformations, infrastructure, and configuration changes.
- Define validation stages for unit tests, schema checks, DAG integrity, data contract enforcement, and environment-specific integration tests.
- Support safe deployment patterns for Airflow DAGs, Spark Structured Streaming jobs, and dbt incremental models with minimal downtime.
- Include promotion controls such as branch strategy, artifact versioning, approvals, canary releases, and rollback mechanisms.
- Ensure idempotent re-runs, reproducible builds, and controlled backfills after deployment.
- Specify how secrets, environment configs, and infrastructure changes are managed.
- Propose monitoring and release health checks that detect both system failures and silent data quality issues.
Constraints
- Use AWS-native infrastructure where practical, but keep the design tool-agnostic enough for client portability.
- Team size is 6 data engineers and 1 platform engineer; operational overhead must stay low.
- Production datasets include regulated client data; no direct production testing with live PII.
- Monthly platform budget for CI/CD and observability additions is capped at $18K.
- Existing stack includes Apache Airflow 2.x, dbt Core, Apache Spark on EMR, Amazon EKS, Terraform, GitHub Actions, and Great Expectations.
