Distributed Training with PyTorch DDP | Dataford Interview Questions

Describe how you would implement a distributed training pipeline using PyTorch's DistributedDataParallel (DDP). Implement a function that converts a configuration into a deterministic training-plan summary. Use the configured backend or default to nccl, compute the number of batches using ceiling division, and indicate that checkpointing is rank-zero-only and cleanup is required.

Input: A dictionary containing world_size, epochs, dataset_size, batch_size, and optional backend and seed values.

Output: A dictionary containing rank_count, batches_per_epoch, backend, checkpoint_rank_zero_only, and has_cleanup.

Constraints

world_size, epochs, dataset_size, and batch_size are positive integers.
dataset_size and batch_size are at most 10^9.
backend, when present, is a string such as nccl or gloo.
The output must contain exactly five summary fields.

Function Signature

def build_ddp_training_plan(config):

Your question is Distributed Training with PyTorch DDP. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Input: A dictionary containing world_size, epochs, dataset_size, batch_size, and optional backend and seed values.

Output: A dictionary containing rank_count, batches_per_epoch, backend, checkpoint_rank_zero_only, and has_cleanup.

Constraints

world_size, epochs, dataset_size, and batch_size are positive integers.
dataset_size and batch_size are at most 10^9.
backend, when present, is a string such as nccl or gloo.
The output must contain exactly five summary fields.

Function Signature

def build_ddp_training_plan(config):

Your question is Distributed Training with PyTorch DDP. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.