Streaming Tokenization Under Tight Memory

MediumPython00:00

Practice interviewer

In session

5 left

00:00

Your question is Streaming Tokenization Under Tight Memory. Start with the requirements on the right.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

You need to log in / sign up to run or submit.

Problem

Write a function to process and tokenize streaming text data under tight memory constraints.

Implement tokenize_stream(chunks), where chunks is an iterable of strings. Return a list of tokens in order, splitting on any whitespace character. Chunks may split a token, and a token may continue across empty chunks. Process the input incrementally rather than concatenating all chunks first.

Contract

Input: an iterable of strings.
Output: a list of non-empty strings.
Use str.isspace() to identify delimiters.

Constraints

Total input length is at most 10^7 characters
chunks is an iterable of strings
Chunks may be empty
Whitespace is defined by str.isspace()
The final token may have no trailing delimiter

Function Signature

def tokenize_stream(chunks):

Your solutionPython 3

You need to log in / sign up to run or submit.

Run your code to see test output