Context
DevFlow, a SaaS developer platform, runs a centralized CI/CD system for 250 engineers across 120 microservices. Today, GitHub Actions triggers full test and deployment workflows on every pull request, but queue times, flaky integration tests, and poor pipeline visibility are slowing releases and frustrating developers.
You are asked to redesign the CI/CD data and orchestration pipeline with empathy for developer workflows: minimize waiting, preserve fast feedback, and provide clear failure signals without sacrificing reliability or compliance.
Scale Requirements
- Repositories: 120 services, 1 monorepo plus 80 standalone repos
- Build volume: 18,000 pipeline runs/day, 250 concurrent jobs at peak
- Artifacts: ~4 TB/month of build logs, test reports, and container metadata
- Latency targets: PR validation < 10 minutes P95, deployment pipeline < 20 minutes P95
- Freshness: Pipeline telemetry available for dashboards within 1 minute
- Retention: Logs for 90 days, audit records for 1 year
Requirements
- Design a pipeline that ingests CI/CD events from GitHub Actions, Jenkins, and Argo CD into a central analytics store.
- Model workflow stages so teams can identify bottlenecks by repo, branch, job type, and engineer.
- Support dependency-aware execution so only impacted services run tests and builds.
- Detect flaky tests, repeated retries, queue buildup, and failed deployments.
- Provide near-real-time dashboards and alerts for developer experience metrics.
- Ensure idempotent ingestion and backfill support for missed webhook events.
- Include monitoring, failure recovery, and data quality checks.
Constraints
- Infrastructure must remain AWS-based and reuse existing GitHub Actions runners and EKS clusters.
- Budget increase is capped at $15K/month.
- Auditability is required for SOC 2; deployment events cannot be lost.
- Team size is 3 data engineers and 2 platform engineers, so operational complexity should stay moderate.
