Identify Frequent Duplicates in Large Dataset | Dataford Interview Questions

If 30% of a billion-record dataset contains duplicate transaction IDs, how would you efficiently identify the most frequent duplicates and optimize the Spark job?

Asked in the R2 PySpark stage. Write a Python function that takes a list of transaction IDs and returns the IDs with the highest duplicate counts, sorted by count descending then ID ascending.

Function Input

transaction_ids: list of strings or integers

Output

list of the most frequent duplicate IDs

Requirements

Count only IDs that appear more than once.
Return every ID tied for the maximum duplicate frequency.
Optimize for large input by avoiding unnecessary full sorts when possible.

Constraints

1 <= len(transaction_ids) <= 10^6
transaction_ids may contain strings or integers
Duplicate frequency is defined as total occurrences greater than 1
Return all IDs tied for the maximum duplicate frequency

Function Signature

def most_frequent_duplicates(transaction_ids):

Your question is Identify Frequent Duplicates in Large Dataset. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem