Context
FinCore, a mid-sized fintech company, runs daily and hourly ETL pipelines that move payment, customer, and audit data from PostgreSQL, S3, and internal APIs into Snowflake. Today, pipeline code is deployed through a basic CI/CD process with shared service accounts, plaintext environment variables in some jobs, and limited auditability; the platform team wants a more secure DevOps model without slowing delivery.
You are asked to design a secure data pipeline platform that integrates DevOps controls directly into build, deployment, orchestration, and runtime operations.
Scale Requirements
- Pipelines: 120 scheduled batch jobs and 15 near-real-time ingestion jobs
- Data volume: 4 TB/day processed, 25 TB retained in raw zone
- Throughput: up to 8,000 records/sec for ingestion APIs, 40 concurrent Airflow tasks
- Latency: hourly jobs must complete within 20 minutes; streaming loads within 2 minutes
- Users: 25 engineers across data engineering, analytics engineering, and platform teams
Requirements
- Design a CI/CD workflow for ETL code that enforces security scans before deployment.
- Protect secrets, credentials, and connection strings used by Airflow, dbt, and Spark jobs.
- Implement least-privilege access across source systems, orchestration, storage, and warehouse layers.
- Ensure data is encrypted in transit and at rest across S3, Kafka, and Snowflake.
- Add controls for artifact signing, dependency vulnerability scanning, and infrastructure-as-code review.
- Define audit logging, monitoring, and incident response for unauthorized access or pipeline tampering.
- Preserve deployment speed: standard pipeline changes should still reach production in under 30 minutes.
Constraints
- AWS is the primary cloud; existing stack includes Airflow 2.x, dbt, Snowflake, S3, and Terraform.
- Compliance requirements include SOC 2 and PCI-adjacent controls for payment metadata.
- Budget allows managed services, but no full platform rewrite.
- Team has limited dedicated security engineering support, so controls should be automatable and operationally simple.
