Transformer Self-Attention From Scratch | Dataford Interview Questions

Implement a basic version of a transformer's self-attention mechanism from scratch.

Implement scaled dot-product attention for query, key, and value matrices represented as nested lists. Return the attention output, applying an optional boolean mask where False entries are excluded. Use a numerically stable softmax and scale scores by the key dimension.

Constraints

1 <= len(query) <= 256
1 <= len(key) == len(value) <= 256
1 <= len(query[0]) == len(key[0]) <= 128
1 <= len(value[0]) <= 128
All rows in each matrix have consistent dimensions
If provided, mask has shape len(query) by len(key) and contains booleans
At least one position in every query mask row is True

Function Signature

def self_attention(query, key, value, mask=None):

Your question is Transformer Self-Attention From Scratch. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Implement a basic version of a transformer's self-attention mechanism from scratch.

Constraints

1 <= len(query) <= 256
1 <= len(key) == len(value) <= 256
1 <= len(query[0]) == len(key[0]) <= 128
1 <= len(value[0]) <= 128
All rows in each matrix have consistent dimensions
If provided, mask has shape len(query) by len(key) and contains booleans
At least one position in every query mask row is True

Function Signature

def self_attention(query, key, value, mask=None):

Your question is Transformer Self-Attention From Scratch. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.