Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Choose Kubernetes Workload for Pipelines

MediumPipelines00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Context

NovaRetail runs Airflow, Spark batch jobs, and a small fleet of Kafka consumers on Amazon EKS. The platform team currently deploys all pipeline components as generic Kubernetes Deployments, which has caused issues with stable identities for stateful workers, duplicate processing during restarts, and inconsistent node-level log collection.

You need to explain when to use Deployments, StatefulSets, and DaemonSets in this data platform, and propose how each should be applied to pipeline workloads.

Scale Requirements

  • Cluster size: 60 EKS worker nodes across 3 AZs
  • Airflow workloads: 2 schedulers, 20-200 ephemeral task pods/day
  • Streaming consumers: 48 Kafka partitions, target consumer lag < 30 seconds
  • Stateful services: 3 metadata/cache replicas with persistent volumes
  • Node-level agents: 1 pod per node for logs and metrics
  • Availability target: 99.9% for orchestration and ingestion services

Requirements

  1. Describe the operational differences between Deployments, StatefulSets, and DaemonSets in Kubernetes.
  2. Map each controller to concrete data engineering workloads such as Airflow webserver/scheduler, Kafka consumers, metadata databases, and node-level observability agents.
  3. Explain implications for scaling, pod identity, storage, rolling updates, and failure recovery.
  4. Show how you would deploy at least one stateless service and one node-level agent on EKS.
  5. Define monitoring and alerting for rollout failures, pod churn, unavailable replicas, and node coverage.
  6. Discuss trade-offs if the team wants to minimize operational complexity while preserving reliability for ETL and streaming jobs.

Constraints

  • AWS-first stack: EKS, EBS, CloudWatch, Prometheus, Grafana
  • Small platform team: 3 engineers supporting all data infrastructure
  • Budget-sensitive: avoid overprovisioning dedicated nodes
  • Compliance: production logs must be collected from every node and retained for 30 days
  • Existing workloads cannot tolerate more than 5 minutes of orchestration downtime