Diagnose Post Migration Pipeline Slowdown

HardPipelines00:00

Practice interviewer

In session

5 left

00:00

Your question is Diagnose Post Migration Pipeline Slowdown. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

You need to log in / sign up to chat or submit.

Problem

Scenario

You have taken over a data pipeline after a system migration and the team has noticed slower processing. Jobs still complete, but overall throughput is down and downstream reporting is delayed. You need to isolate whether the regression comes from ingestion, transformation logic, orchestration, infrastructure, or data quality checks.

Question

A client is seeing a 20% drop in data processing speed after a system migration. How would you diagnose and fix the issue?

What to Inspect First

Compare pre migration and post migration runtime by stage

Check infrastructure sizing, concurrency, and resource saturation

Inspect Airflow queue delays, retries, and dependency waits

Review Spark skew, shuffle volume, and executor memory pressure

Validate whether new data quality checks or schema changes added latency