Tokenization for Transformer Models | Dataford Interview Questions

Write a function to process and tokenize text data for a transformer-based model.

Implement process_and_tokenize(text, vocab, max_length) using lowercase word and punctuation tokenization. Add [CLS] and [SEP], map unknown tokens to [UNK], truncate while preserving [SEP], and right-pad with [PAD].

Return a dictionary containing input_ids and attention_mask, both of length max_length. The vocabulary contains integer IDs for all four special tokens.

Constraints

0 <= len(text) <= 5000
3 <= max_length <= 512
vocab contains integer IDs for [PAD], [UNK], [CLS], and [SEP]
Vocabulary keys are lowercase tokens or special-token names
Return both arrays with exactly max_length elements

Function Signature

def process_and_tokenize(text, vocab, max_length):

Your question is Tokenization for Transformer Models. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Write a function to process and tokenize text data for a transformer-based model.

Return a dictionary containing input_ids and attention_mask, both of length max_length. The vocabulary contains integer IDs for all four special tokens.

Constraints

0 <= len(text) <= 5000
3 <= max_length <= 512
vocab contains integer IDs for [PAD], [UNK], [CLS], and [SEP]
Vocabulary keys are lowercase tokens or special-token names
Return both arrays with exactly max_length elements

Function Signature

def process_and_tokenize(text, vocab, max_length):

Your question is Tokenization for Transformer Models. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.