Your question is Handling Data Skew in Pipelines. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
How do you handle data skew in a data processing pipeline?
Explain how you would detect skewed keys, identify whether the problem occurs during partitioning, joins, aggregation, or file output, and select an appropriate mitigation. Address salting, adaptive query execution, repartitioning, broadcast joins, key splitting, and pre-aggregation. Describe how your approach preserves correctness, handles hot keys that change over time, supports batch and streaming workloads, and avoids creating a new bottleneck. Include testing, backfill, observability, and rollback considerations using a stack such as Apache Spark, Apache Kafka, and a cloud data lake or warehouse.