Context
A data platform operates regulated customer data pipelines across AWS, Azure, and GCP. Today, ingestion and transformation jobs use broad cloud IAM roles and shared service principals, which creates audit risk, over-permissioned access to storage, and weak isolation between dev, staging, and prod.
You are asked to design a least-privilege access model for data pipelines that run batch and streaming workloads using managed declarative pipelines, workflow orchestration, a centralized data catalog, and cloud-native identities. The design must support secure cross-cloud ingestion, environment isolation, and auditable access to bronze, silver, and gold data products.
Scale Requirements
- Pipelines: 1,200 scheduled batch pipelines and 180 continuous streaming pipelines
- Data volume: 9 PB total in object storage, 45 TB/day new data
- Tenancy: 60 internal platform teams, 300+ service identities, 3 environments per cloud
- Latency: streaming SLA < 3 minutes end-to-end; batch completion by 6 AM local region
- Auditability: all access decisions traceable within 15 minutes
Requirements
- Design identity and access boundaries for data platform workspaces, centralized data catalogs, catalogs, schemas, tables, volumes, and external locations across AWS, Azure, and GCP.
- Enforce least privilege for pipeline execution identities so each pipeline can read only required sources and write only approved targets.
- Support secretless authentication where possible using AWS IAM roles, Azure Managed Identities, and GCP service accounts with cloud credential passthrough equivalents or workload identity patterns.
- Define how workflow orchestrators and declarative data pipelines obtain short-lived credentials for cloud storage and downstream systems.
- Include controls for schema evolution, data quality failures, and restricted promotion from bronze to silver/gold.
- Provide monitoring, alerting, and automated remediation for privilege drift, failed policy enforcement, and unauthorized access attempts.
Constraints
- Must use platform-native governance first: centralized data catalogs, service principals, cluster policies, compute policies, and audit logs.
- No long-lived static cloud keys stored in notebooks or pipeline configs.
- Must satisfy SOC 2, GDPR, and customer-managed VPC/VNet deployment patterns.
- Incremental platform budget increase is capped at $40K/month.