Your question is Explain Text Tokenization. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
You are working on an NLP pipeline and need to convert raw text into units a model can process. Before training or using a language model, you need to decide how text should be split and represented.
What is tokenization?
Tokenization is the bridge between raw strings and model inputs. It affects vocabulary coverage, sequence length, handling of rare words, and downstream model quality.