Preprocess Text for LLMs | Dataford Interview Questions

Write a function to preprocess textual data for a language model.

Implement preprocess_text(text, max_tokens) to lowercase text, extract words, numbers, and contractions, replace URLs with <URL>, and return at most max_tokens tokens. Return an empty list when max_tokens is zero or negative. For example, "We're ready!" produces ["we're", "ready"], and "See https://xometry.com/quote." produces ["see", "<URL>"].

Constraints

0 <= len(text) <= 10000
-1000 <= max_tokens <= 1000
Tokens contain only ASCII letters and digits, with an optional internal apostrophe
URLs use the http:// or https:// scheme
Return tokens in their original order

Function Signature

def preprocess_text(text, max_tokens):

Your question is Preprocess Text for LLMs. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Write a function to preprocess textual data for a language model.

Constraints

0 <= len(text) <= 10000
-1000 <= max_tokens <= 1000
Tokens contain only ASCII letters and digits, with an optional internal apostrophe
URLs use the http:// or https:// scheme
Return tokens in their original order

Function Signature

def preprocess_text(text, max_tokens):

Your question is Preprocess Text for LLMs. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.