Diagnosing Spark Data Skew | Dataford Interview Questions

How do you diagnose and resolve data skew in a Spark cluster when processing multi-terabyte datasets?

Implement plan_skew_mitigation(partition_loads, target_ratio) to model this decision algorithmically. Given an array of nonnegative partition loads and an integer target ratio, use the lower median load as the baseline and return an array of split factors. Each factor must be the minimum number of tasks needed to keep that partition at or below target_ratio * median, with a minimum threshold of 1 when the median is zero.

Constraints

1 <= partition_loads.length <= 1000
0 <= partition_loads[i] <= 10^12
1 <= target_ratio <= 1000
The input contains at least one partition

Function Signature

def plan_skew_mitigation(partition_loads, target_ratio):

Your question is Diagnosing Spark Data Skew. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

How do you diagnose and resolve data skew in a Spark cluster when processing multi-terabyte datasets?

Constraints

1 <= partition_loads.length <= 1000
0 <= partition_loads[i] <= 10^12
1 <= target_ratio <= 1000
The input contains at least one partition

Function Signature

def plan_skew_mitigation(partition_loads, target_ratio):

Your question is Diagnosing Spark Data Skew. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.