PySpark Output Schema | Dataford Interview Questions

Given links to production-level PySpark code (150+ lines, pre-processing and processing), write the schema of the output file from each code and give an example of the schema and data after the code runs.

Asked in the Round 3 stage. For this coding version, represent the pipeline as ordered transformation objects and process rows in memory. Return the final schema and resulting rows.

Function: derive_output(source_schema, rows, operations)

source_schema is a list of {name, type} objects. Supported operations are select, withColumn using multiply, filter using eq or gt, and groupBy using sum or count. Preserve column order.

Constraints

1 <= len(source_schema) <= 50
0 <= len(rows) <= 1000
0 <= len(operations) <= 50
Column names referenced by operations exist at that point in the pipeline
Each groupBy operation contains exactly one aggregation

Function Signature

def derive_output(source_schema, rows, operations):

Your question is PySpark Output Schema. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem