Ring-AllReduce Scaling | Dataford Interview Questions

Explain the mechanics of the Ring-AllReduce algorithm and how it scales as you increase the number of GPU nodes for distributed training on AMD Instinct.

Implement ring_all_reduce(tensors) to simulate reduce-scatter followed by all-gather for equal-length GPU tensors. Return the element-wise sum replicated on every node, along with the number of communication rounds and point-to-point messages.

Constraints

1 <= len(tensors) <= 64
Each tensor has the same length
len(tensors[0]) is positive and divisible by len(tensors)
Each tensor element is an integer between -10^6 and 10^6
The result must preserve the original tensor length

Function Signature

def ring_all_reduce(tensors):

Your question is Ring-AllReduce Scaling. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Explain the mechanics of the Ring-AllReduce algorithm and how it scales as you increase the number of GPU nodes for distributed training on AMD Instinct.

Constraints

1 <= len(tensors) <= 64
Each tensor has the same length
len(tensors[0]) is positive and divisible by len(tensors)
Each tensor element is an integer between -10^6 and 10^6
The result must preserve the original tensor length

Function Signature

def ring_all_reduce(tensors):

Your question is Ring-AllReduce Scaling. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.