Transformer-Based Machine Learning Coding | Dataford Interview Questions

How would you implement a machine learning coding solution related to transformers?

Implement the scaled dot-product attention operation used in transformer models. Given query, key, and value matrices, return the attention output, optionally ignoring positions marked false by a boolean mask. Use pure code and do not rely on machine learning libraries.

Input/output: query is [query_length][depth], key is [key_length][depth], value is [key_length][value_depth], and mask is either None or [query_length][key_length]. Return [query_length][value_depth].

Constraints

query, key, and value contain at least one row
len(query[0]) == len(key[0])
len(key) == len(value)
mask is None or has shape [len(query)][len(key)]
Every query row has at least one unmasked key
All matrix entries are finite real numbers

Function Signature

def scaled_dot_product_attention(query, key, value, mask):

Your question is Transformer-Based Machine Learning Coding. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

How would you implement a machine learning coding solution related to transformers?

Constraints

query, key, and value contain at least one row
len(query[0]) == len(key[0])
len(key) == len(value)
mask is None or has shape [len(query)][len(key)]
Every query row has at least one unmasked key
All matrix entries are finite real numbers

Function Signature

def scaled_dot_product_attention(query, key, value, mask):

Your question is Transformer-Based Machine Learning Coding. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.