Your question is Optimize Slow Spark Pipelines. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
How would you optimize a slow-running Spark pipeline handling large distributed data sets? Explain how you would diagnose the bottleneck, evaluate the query plan and Spark UI, optimize joins and partitioning, address data skew, tune memory and parallelism, and validate that changes improve runtime without compromising correctness.
