Drop Duplicates in PySpark | Dataford Interview Questions

In PySpark, drop duplicates while retaining the latest record based on a timestamp column.

Asked in the technical stage. Implement the equivalent transformation as a Python function over a list of record dictionaries.

Input and Output

records is a list of dictionaries containing the fields in key_columns and an integer timestamp. Return one record per unique key, retaining the record with the greatest timestamp. If timestamps tie, retain the earliest input record. Return retained records ordered by descending timestamp, preserving input order for ties.

Constraints

0 <= records.length <= 10^4
1 <= key_columns.length
Every key column exists in every record
Each record contains an integer timestamp
If timestamps tie within a key, retain the earliest input record
Return records in descending timestamp order

Function Signature

def retain_latest_records(records, key_columns):

Your question is Drop Duplicates in PySpark. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem