Tokenization and Embedding Batch Optimization | Dataford Interview Questions

Optimize a script that processes large batches of text for tokenization and embedding generation. Implement the function below so repeated texts are tokenized and embedded once, processing respects the maximum batch token count, and results remain aligned with the original input order.

Contract

texts is a list of strings, token_vectors maps lowercase tokens to equal-length numeric vectors, and max_tokens is a positive batch limit. Return one averaged embedding per input text, using a zero vector when no token has a known vector.

Constraints

1 <= len(texts) <= 5000
1 <= max_tokens
token_vectors contains at least one token vector
All token vectors have the same positive dimension
Each input text contains at most 10000 whitespace-separated tokens
Token matching is lowercase and whitespace-based

Function Signature

def generate_embeddings(texts, token_vectors, max_tokens):

Your question is Tokenization and Embedding Batch Optimization. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Contract

Constraints

1 <= len(texts) <= 5000
1 <= max_tokens
token_vectors contains at least one token vector
All token vectors have the same positive dimension
Each input text contains at most 10000 whitespace-separated tokens
Token matching is lowercase and whitespace-based

Function Signature

def generate_embeddings(texts, token_vectors, max_tokens):

Your question is Tokenization and Embedding Batch Optimization. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.