Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Design Production Observability Pipeline

HardPipelines00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Context

Meta operates a large-scale production service with telemetry emitted from service binaries, edge layers, and infrastructure hosts. The current setup relies on fragmented log and metric collection with service-local alerts, which causes delayed incident detection, inconsistent schemas, and high alert noise during regional failures.

Design a centralized observability and alerting pipeline using Meta-preferred surfaces such as Scribe for log transport, Scuba for interactive analytics, and internal service orchestration hooks. The goal is to ingest, validate, aggregate, and route telemetry into real-time alerting and longer-term analytical storage.

Scale Requirements

  • Telemetry volume: 15M events/second peak across logs, metrics, and traces
  • Payload size: 0.8-2.5 KB/event, mixed JSON and protobuf
  • Daily ingest: ~1.2 PB raw telemetry
  • Latency target: P95 under 30 seconds from emission to alert evaluation
  • Retention: 30 days hot, 180 days cold archive
  • Availability target: 99.99% for ingestion and alert evaluation

Requirements

  1. Build a unified ingestion pipeline for logs, metrics, and trace-derived events from thousands of services.
  2. Enforce schema validation, deduplication, tenant/service tagging, and PII redaction before downstream storage.
  3. Support both streaming alert evaluation and batch backfills for rule changes or incident replay.
  4. Route curated telemetry to low-latency serving for dashboards and alerts, plus cost-efficient long-term storage.
  5. Provide idempotent reprocessing, replay from checkpoints, and dead-letter handling for malformed or oversized events.
  6. Design monitoring for pipeline health, data quality, alert freshness, and false-positive spikes.
  7. Support multi-region failover without losing more than 60 seconds of data.

Constraints

  • Must integrate with existing Meta telemetry producers and on-call workflows.
  • Cross-region replication cost must be controlled; not all raw data can stay in hot storage.
  • PII and security-sensitive fields must be masked before analyst-facing storage.
  • Rule updates should deploy without stopping ingestion or causing duplicate alerts.