Your question is Implement Multi-Head Attention. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Implement multi-head self-attention using TensorFlow or PyTorch, focusing on the Google source code.
Asked in the round1 stage. Followed by questions on DeBERTa's disentangled attention/enhanced mask decoder and pre-norm vs post-norm.
Address the tensor shapes, query/key/value projections, attention scaling, masking, head splitting and merging, numerical stability, and computational complexity. Provide a working implementation, explain the major design choices, and describe how you would test correctness against a trusted reference implementation.