Context

FinCore, a mid-sized fintech company, runs daily and hourly ETL pipelines that move payment, customer, and audit data from PostgreSQL, S3, and internal APIs into Snowflake. Today, pipeline code is deployed through a basic CI/CD process with shared service accounts, plaintext environment variables in some jobs, and limited auditability; the platform team wants a more secure DevOps model without slowing delivery.

You are asked to design a secure data pipeline platform that integrates DevOps controls directly into build, deployment, orchestration, and runtime operations.

Scale Requirements

• Pipelines: 120 scheduled batch jobs and 15 near-real-time ingestion jobs
• Data volume: 4 TB/day processed, 25 TB retained in raw zone
• Throughput: up to 8,000 records/sec for ingestion APIs, 40 concurrent Airflow tasks
• Latency: hourly jobs must complete within 20 minutes; streaming loads within 2 minutes
• Users: 25 engineers across data engineering, analytics engineering, and platform teams

Requirements

1. Design a CI/CD workflow for ETL code that enforces security scans before deployment.
2. Protect secrets, credentials, and connection strings used by Airflow, dbt, and Spark jobs.
3. Implement least-privilege access across source systems, orchestration, storage, and warehouse layers.
4. Ensure data is encrypted in transit and at rest across S3, Kafka, and Snowflake.
5. Add controls for artifact signing, dependency vulnerability scanning, and infrastructure-as-code review.
6. Define audit logging, monitoring, and incident response for unauthorized access or pipeline tampering.
7. Preserve deployment speed: standard pipeline changes should still reach production in under 30 minutes.

Constraints

• AWS is the primary cloud; existing stack includes Airflow 2.x, dbt, Snowflake, S3, and Terraform.
• Compliance requirements include SOC 2 and PCI-adjacent controls for payment metadata.
• Budget allows managed services, but no full platform rewrite.
• Team has limited dedicated security engineering support, so controls should be automatable and operationally simple.