Your question is PySpark Aggregation at Scale. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Can you write a PySpark script to aggregate a large dataset distributed across multiple nodes? Explain how you would read the source data, define the aggregation, handle malformed records, and write the result efficiently. Your solution should address partitioning, data skew, null values, job observability, and safe reruns.