Diagnose Pipeline Performance Drop

A customer reports a sudden drop in performance in a production data pipeline. You need to diagnose whether the slowdown is coming from ingestion, stream processing, storage, orchestration, or downstream transformations, and determine the fastest path to isolate the issue.

Problem

What to Inspect

Pipeline latency by stage
Kafka lag and partition skew
Spark batch duration and executor pressure
Snowflake load and query queue time
Airflow task failures, retries, and SLA misses
DLQ growth and schema validation failures

Reference Stack

Storage·Amazon S3 Parquet + SnowflakeIngestion·Bayone Solutions Ingestion API + Apache Kafka 3.xProcessing·Apache Spark Structured Streaming 3.5Transforms·dbt Core 1.7Observability·Prometheus + GrafanaOrchestration·Apache Airflow 2.8

Problem

What to Inspect

Pipeline latency by stage
Kafka lag and partition skew
Spark batch duration and executor pressure
Snowflake load and query queue time
Airflow task failures, retries, and SLA misses
DLQ growth and schema validation failures

Reference Stack

Problem

What to Inspect

Pipeline latency by stage
Kafka lag and partition skew
Spark batch duration and executor pressure
Snowflake load and query queue time
Airflow task failures, retries, and SLA misses
DLQ growth and schema validation failures

Interview Guides

Problem

What to Inspect

Reference Stack

Problem

What to Inspect

Reference Stack

Diagnose Pipeline Performance Drop

Problem

What to Inspect

Reference Stack

Problem

What to Inspect

Reference Stack