Dataford
Interview QuestionsInterview GuidesExperiencesMock InterviewsPricing
Get started

Diagnose Pipeline Performance Drop

MediumPipelines00:00
I
Practice interviewer
In session
5 left
00:00

Your question is Diagnose Pipeline Performance Drop. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

You need to log in / sign up to chat or submit.

Problem

A customer reports a sudden drop in performance in a production data pipeline. You need to diagnose whether the slowdown is coming from ingestion, stream processing, storage, orchestration, or downstream transformations, and determine the fastest path to isolate the issue.

What to Inspect

  • Pipeline latency by stage
  • Kafka lag and partition skew
  • Spark batch duration and executor pressure
  • Snowflake load and query queue time
  • Airflow task failures, retries, and SLA misses
  • DLQ growth and schema validation failures

Reference Stack

Storage·Amazon S3 Parquet + SnowflakeIngestion·Bayone Solutions Ingestion API + Apache Kafka 3.xProcessing·Apache Spark Structured Streaming 3.5Transforms·dbt Core 1.7Observability·Prometheus + GrafanaOrchestration·Apache Airflow 2.8