Context
FinSight, a B2B fintech analytics company, runs batch and near-real-time data pipelines on AWS using Airflow, dbt, Spark, and Snowflake. The team frequently sees issues that pass in development but fail in staging or production because of inconsistent infrastructure, package versions, secrets, schemas, and orchestration settings.
You need to design a deployment and runtime strategy that enforces environment parity across development, staging, and production while still allowing safe testing and controlled releases.
Scale Requirements
- Pipelines: 120 Airflow DAGs, 40 dbt models, 15 Spark jobs
- Data volume: 6 TB/day batch, 25K events/sec streaming peak
- Latency: staging validation within 15 minutes of deploy; production rollback within 10 minutes
- Team usage: 18 data engineers deploying changes daily
- Retention: 180 days raw data, 2 years curated warehouse data
Requirements
- Design a promotion workflow so code, infrastructure, and configuration move consistently from development to staging to production.
- Ensure the same runtime dependencies, container images, Airflow versions, dbt packages, and Spark libraries are used across environments.
- Define how environment-specific values such as secrets, warehouse sizes, S3 buckets, and topic names are injected without changing application code.
- Add automated checks for schema compatibility, data quality, and DAG health before promotion.
- Support reproducible local development for engineers and deterministic CI/CD builds.
- Include rollback, auditability, and change approval for production releases.
- Explain how you would validate parity for both batch jobs and streaming consumers.
Constraints
- AWS is the primary cloud; Snowflake is already in use and cannot be replaced.
- Production data cannot be copied to development except for masked samples.
- Budget allows limited always-on staging compute; ephemeral test environments are preferred.
- SOX-style audit requirements apply to production changes.
- The team wants to minimize manual release steps and avoid environment-specific code branches.
