Debugging a PySpark Pipeline | Dataford Interview Questions

Can you walk through your process for debugging a complex PySpark pipeline? For the coding portion, implement first_bad_stage(stage_outputs, required_fields). stage_outputs is an ordered array of stage objects, each containing a name and output rows; required_fields maps each stage name to required field names. Return [stage_index, row_index] for the first row that is missing a required field or contains a null value, scanning stages and rows in order, or [-1, -1] when all outputs are valid.

Constraints

1 <= stage_outputs.length <= 10^4
Each stage name is unique
Each stage contains zero or more row objects
Each required field name is a string
A row is valid when every required field exists and has a non-null value

Function Signature

def first_bad_stage(stage_outputs, required_fields):

Your question is Debugging a PySpark Pipeline. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Constraints

1 <= stage_outputs.length <= 10^4
Each stage name is unique
Each stage contains zero or more row objects
Each required field name is a string
A row is valid when every required field exists and has a non-null value

Function Signature

def first_bad_stage(stage_outputs, required_fields):

Your question is Debugging a PySpark Pipeline. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.