Your question is Diagnose Databricks Pipeline Bottlenecks. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Databricks runs a production lakehouse pipeline that ingests application logs and CDC data into Delta tables using Databricks Workflows, Auto Loader, and Delta Live Tables. Over the last two weeks, several jobs have missed their SLA, but Spark UI symptoms are inconsistent: some runs show executor under-utilization, others stall during file discovery, checkpoint commits, or downstream table writes.
You are asked to design an operational debugging approach for a DevOps Engineer supporting Databricks pipelines. The goal is not only to identify whether the bottleneck is CPU, memory, disk, file descriptors, or network, but also to connect Linux-level signals from tools such as top, strace, lsof, tcpdump, and iostat to Databricks-specific pipeline stages and remediation actions.
top, strace, lsof, tcpdump, and iostat on driver and worker nodes.