Spark Skew Optimization | Dataford Interview Questions

How do you optimize a Spark job that is suffering from data skew?

Implement a deterministic Python simulation of salting. Given record keys, known hot keys, partition count, and salt bucket count, return the number of records assigned to each partition. Non-hot keys use sum(ord(c) for c in key) % num_partitions; occurrences of hot keys rotate through salt buckets before mapping to partitions.

Function Contract

def partition_loads(records, hot_keys, num_partitions, salt_buckets): returns a list of num_partitions integers.

Constraints

1 <= len(records) <= 10^4
0 <= len(hot_keys) <= len(records)
Each record key is a non-empty string
1 <= num_partitions <= 10^3
1 <= salt_buckets <= 10^3

Function Signature

def partition_loads(records, hot_keys, num_partitions, salt_buckets):

Your question is Spark Skew Optimization. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem