Python Memory Management and Spark Session | Dataford Interview Questions

How do you manage Python memory effectively, and what do you consider when initializing a Spark session?

Implement a planner that converts workload and executor inputs into a safe Spark configuration. It must reserve Python memory overhead, avoid oversized batches, choose a practical partition count, and represent dynamic allocation settings.

I/O Contract

Implement create_spark_plan(data_size_mb, executor_memory_mb, executor_cores, python_memory_mb, dynamic_allocation). Return a dictionary containing the calculated memory, partition, batch, and Spark configuration values.

Constraints

0 <= data_size_mb
executor_memory_mb > 0
executor_cores > 0
python_memory_mb > 0
Use a 128 MB target partition size
Minimum memory overhead is 384 MB

Function Signature

def create_spark_plan(data_size_mb, executor_memory_mb, executor_cores, python_memory_mb, dynamic_allocation):

Your question is Python Memory Management and Spark Session. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem