Transformer Block in PyTorch | Dataford Interview Questions

Implement a transformer block from scratch in PyTorch.

Implement the forward pass for a batch-first transformer block with multi-head self-attention, scaled dot-product attention, residual connections, layer normalization, and a two-layer feed-forward network. For grading, represent x as a nested Python list with shape [batch_size][sequence_length][d_model]; return a nested list with the same shape. Use dropout = 0 for deterministic execution. The reference implementation uses identity projections to keep the standard-library test harness deterministic.

Constraints

1 <= batch_size <= 64
1 <= sequence_length <= 512
1 <= d_model <= 1024
d_model is divisible by num_heads
1 <= num_heads <= d_model
1 <= d_ff
dropout = 0 for deterministic grading
x contains finite floating-point values

Function Signature

def transformer_block(x, d_model, num_heads, d_ff, dropout):

Your question is Transformer Block in PyTorch. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Implement a transformer block from scratch in PyTorch.

Constraints

1 <= batch_size <= 64
1 <= sequence_length <= 512
1 <= d_model <= 1024
d_model is divisible by num_heads
1 <= num_heads <= d_model
1 <= d_ff
dropout = 0 for deterministic grading
x contains finite floating-point values

Function Signature

def transformer_block(x, d_model, num_heads, d_ff, dropout):

Your question is Transformer Block in PyTorch. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.