From-Scratch Transformer | Dataford Interview Questions

Implement a transformer from scratch.

Asked in the later round coding stage. OP said the coding part was mainly hand-writing a transformer.

Implement transformer_forward(tokens, params) using one encoder block. params contains an embedding matrix, Q/K/V/output projection matrices, feed-forward matrices, layer-normalization parameters, and num_heads. Use full self-attention, scale scores by the square root of the head dimension, apply row-wise softmax, then apply residual connections and post-layer normalization around both the attention and feed-forward sublayers. Use ReLU in the feed-forward network.

The function returns a list of vectors, one per input token. Matrices use output-row orientation: output[i] = sum(matrix[i][j] * vector[j]). Use epsilon 1e-5 in layer normalization.

Signature: transformer_forward(tokens, params)

Constraints: the model dimension is divisible by num_heads, and all parameter dimensions are valid.

Constraints

1 <= len(tokens)
1 <= len(embedding)
The model dimension is positive and divisible by num_heads
All tokens are valid indices into embedding
All matrices have dimensions compatible with the model
Use full, non-causal self-attention
Use epsilon 1e-5 in layer normalization

Function Signature

def transformer_forward(tokens, params):

Your question is From-Scratch Transformer. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem