Autoloader in Databricks | Dataford Interview Questions

Write code to implement Autoloader in Databricks. Asked in the Technical interview stage. Implement autoload_file_batches(batches), where batches is a list of file batches. Each file has a unique path identifier and a list of dictionary records. Process files in order, ignore repeated paths, and return all records from newly discovered files. The output must include every field seen across all records, using None when a record lacks a field.

Constraints

0 <= len(batches) <= 100
Each batch contains at most 100 files
Each file contains at most 100 records
Each record is a dictionary with string field names
File paths are strings
Missing fields must be represented by None

Function Signature

def autoload_file_batches(batches):

Your question is Autoloader in Databricks. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Constraints

0 <= len(batches) <= 100
Each batch contains at most 100 files
Each file contains at most 100 records
Each record is a dictionary with string field names
File paths are strings
Missing fields must be represented by None

Function Signature

def autoload_file_batches(batches):

Your question is Autoloader in Databricks. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.