Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Design Data Contracts for Pipelines

HardPipelines00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Context

Meta’s Ads Insights platform ingests campaign performance events from multiple producers, including Ads Delivery, Billing, and conversion reporting systems. Today, downstream Hive and Presto consumers often discover breaking schema changes only after scheduled Airflow backfills or near-real-time Flink jobs fail, causing dashboard delays and inconsistent metrics.

You are asked to design a data contract framework for shared pipelines so producer teams can evolve schemas safely while consumer teams retain predictable SLAs. The goal is to define what a data contract is, who owns it, and how contract enforcement should work across batch and streaming systems.

Scale Requirements

  • Producers: 40+ upstream datasets across Ads and Measurement
  • Volume: 3B events/day, peak 250K events/sec on streaming topics
  • Consumers: 150+ downstream tables, ML features, and reporting jobs
  • Latency: streaming validation under 2 minutes; batch contract checks before hourly publication
  • Retention: 180 days raw, 3 years curated aggregates

Requirements

  1. Define the contents of a data contract for Meta pipelines: schema, semantics, freshness SLA, quality thresholds, ownership, and change policy.
  2. Specify ownership boundaries between producer teams, central data platform, and downstream consumers.
  3. Design enforcement for both streaming ingestion (Kafka/Flink) and batch publication (Hive/Spark).
  4. Support backward-compatible schema evolution, versioning, and deprecation windows.
  5. Prevent bad data from reaching curated datasets using automated validation, quarantine, and rollback paths.
  6. Expose contract status, violations, and lineage in a way that on-call engineers and analysts can act on quickly.
  7. Describe how orchestration should block downstream publishes when contract checks fail, while still allowing controlled overrides.

Constraints

  • Prefer Meta-adjacent infrastructure: Kafka, Apache Flink, Spark, Hive, Presto, Airflow-like orchestration, and internal metadata services.
  • Some producers are low-maturity teams and cannot manually coordinate every schema change.
  • Contract checks must add less than 5% compute overhead to existing pipelines.
  • PII fields require explicit classification and deletion compatibility.

Your answer should explain the contract model, ownership model, enforcement architecture, and operational playbook.