Dataford
Popular roles
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Browse
Browse All RolesEvery role hub, from analyst to MLBrowse All CompaniesCompany-specific interview loopsAll Interview GuidesThe full guide library
Top questions by role
Software EngineerData AnalystData ScientistData EngineerBusiness AnalystAI EngineerMachine Learning EngineerProduct Manager
Top questions by skill
SQLPythonStatisticsMachine LearningA/B TestingSystem DesignGenerative AIProduct SenseMetricsBehavioral
Browse all questions →Try a mock interview
ExperiencesCoaching
New
Practice
Mock InterviewsTimed interview simulations with feedbackSuccess PathYour 6-week structured planModulesCurated lessons by topicWebinarsTalks from ex-Big Tech data leadsPlaygroundA free-form scratch editor
Learn
BlogInterview strategy and career adviceTech Job Market ReportHiring trends across data and AI rolesFor UniversitiesDataford for career centersAbout DatafordWho we are and how we build
Pricing
Get started

Build Splunk Observability Log Pipeline

MediumPipelines00:00
Practice interviewer
In session
5 left
00:00

Whenever you're ready, walk me through how you'd approach this.

You need to log in / sign up to chat or submit.

Problem

Context

PulseCart, a mid-sized e-commerce company, runs customer-facing services on Kubernetes and currently ships application logs, infrastructure metrics, and deployment events into separate tools. The operations team wants a unified observability pipeline centered on Splunk so they can search logs, correlate incidents, detect failures faster, and support postmortems without relying on manual log collection.

You are asked to design a production-grade pipeline that ingests operational telemetry from microservices, databases, and Kubernetes clusters into Splunk for monitoring and troubleshooting, while also preserving raw data in low-cost storage for replay and audit.

Scale Requirements

  • Services: 120 microservices across 3 Kubernetes clusters
  • Log volume: 2 TB/day raw JSON logs
  • Metrics/events: 1.5M metric datapoints/minute, 50K deployment/audit events/hour
  • Peak throughput: 80K log events/second during incidents
  • Latency target: searchable in Splunk within 60 seconds
  • Retention: 30 days hot in Splunk, 180 days archived in S3

Requirements

  1. Ingest logs, metrics, and operational events from Kubernetes, application services, CI/CD, and PostgreSQL into a centralized pipeline.
  2. Normalize records into a common schema with fields such as service, env, cluster, trace_id, severity, and event_time.
  3. Route valid telemetry to Splunk for observability use cases including incident triage, alerting, dashboarding, and root-cause analysis.
  4. Persist raw and rejected records to Amazon S3 for replay, compliance, and backfills.
  5. Implement data quality checks for malformed JSON, missing required fields, duplicate deployment events, and timestamp drift.
  6. Orchestrate batch backfills and replay workflows without interrupting real-time ingestion.
  7. Define monitoring, alerting, and failure recovery for all stages.

Constraints

  • AWS is the required cloud platform.
  • The team already uses Apache Airflow 2.x and Amazon S3.
  • Incremental budget is capped at $18K/month.
  • PII in logs must be masked before indexing into Splunk.
  • The design should avoid vendor lock-in for preprocessing so another sink could be added later.