Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Least-Privilege Multi-Cloud Data Pipelines

HardPipelines00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Context

A data platform operates regulated customer data pipelines across AWS, Azure, and GCP. Today, ingestion and transformation jobs use broad cloud IAM roles and shared service principals, which creates audit risk, over-permissioned access to storage, and weak isolation between dev, staging, and prod.

You are asked to design a least-privilege access model for data pipelines that run batch and streaming workloads using managed declarative pipelines, workflow orchestration, a centralized data catalog, and cloud-native identities. The design must support secure cross-cloud ingestion, environment isolation, and auditable access to bronze, silver, and gold data products.

Scale Requirements

  • Pipelines: 1,200 scheduled batch pipelines and 180 continuous streaming pipelines
  • Data volume: 9 PB total in object storage, 45 TB/day new data
  • Tenancy: 60 internal platform teams, 300+ service identities, 3 environments per cloud
  • Latency: streaming SLA < 3 minutes end-to-end; batch completion by 6 AM local region
  • Auditability: all access decisions traceable within 15 minutes

Requirements

  1. Design identity and access boundaries for data platform workspaces, centralized data catalogs, catalogs, schemas, tables, volumes, and external locations across AWS, Azure, and GCP.
  2. Enforce least privilege for pipeline execution identities so each pipeline can read only required sources and write only approved targets.
  3. Support secretless authentication where possible using AWS IAM roles, Azure Managed Identities, and GCP service accounts with cloud credential passthrough equivalents or workload identity patterns.
  4. Define how workflow orchestrators and declarative data pipelines obtain short-lived credentials for cloud storage and downstream systems.
  5. Include controls for schema evolution, data quality failures, and restricted promotion from bronze to silver/gold.
  6. Provide monitoring, alerting, and automated remediation for privilege drift, failed policy enforcement, and unauthorized access attempts.

Constraints

  • Must use platform-native governance first: centralized data catalogs, service principals, cluster policies, compute policies, and audit logs.
  • No long-lived static cloud keys stored in notebooks or pipeline configs.
  • Must satisfy SOC 2, GDPR, and customer-managed VPC/VNet deployment patterns.
  • Incremental platform budget increase is capped at $40K/month.