Debugging Distributed Training Slowdowns | Dataford Interview Questions

Debug a scenario where a distributed training job is experiencing significant performance degradation.

Implement diagnose_training_degradation(records, threshold) to aggregate timing records, identify ranks whose average step time exceeds the global average by threshold, and classify the dominant bottleneck as compute, communication, or balanced.

Each record is a dictionary with integer step and rank fields plus numeric compute_ms and communication_ms fields. Return a dictionary containing sorted stragglers and the bottleneck classification.

Constraints

1 <= len(records) <= 500
Each record contains step, rank, compute_ms, and communication_ms
step and rank are integers
compute_ms and communication_ms are non-negative numbers
1.0 <= threshold
Every rank has at least one record

Function Signature

def diagnose_training_degradation(records, threshold):

Your question is Debugging Distributed Training Slowdowns. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Debug a scenario where a distributed training job is experiencing significant performance degradation.

Constraints

1 <= len(records) <= 500
Each record contains step, rank, compute_ms, and communication_ms
step and rank are integers
compute_ms and communication_ms are non-negative numbers
1.0 <= threshold
Every rank has at least one record

Function Signature

def diagnose_training_degradation(records, threshold):

Your question is Debugging Distributed Training Slowdowns. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.