PySpark Fundamentals | Dataford Interview Questions

Tell me about the PySpark fundamentals you rely on when building data pipelines.

Implement those fundamentals by filtering invalid records, keeping the highest-version record for each event, aggregating by entity, and sorting the result.

Input is a list of dictionaries. Return dictionaries containing entity, event_count, and total_amount.

Constraints

0 <= len(records) <= 1000
Each record contains entity, event_id, amount, version, and is_valid
entity and event_id are non-empty strings
amount and version are integers
For duplicate event IDs, the record with the greatest version is retained
Return entities in ascending lexicographic order

Function Signature

def summarize_records(records):

Your question is PySpark Fundamentals. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem