Streaming Data Pipeline Optimization | Dataford Interview Questions

Optimize a Python-based data pipeline to handle massive streaming inputs for model fine-tuning.

Implement pack_stream(records, block_size, eos_token). Process records in order, keep only the first record for each id, append eos_token after every non-empty accepted record, and return complete fixed-size token blocks. Discard any incomplete final block. The function must return a list of lists, not a generator.

records is a list of dictionaries with integer id and integer-list tokens; block_size is a positive integer. Preserve token order and do not add an EOS token for empty or duplicate records.

Constraints

1 <= block_size
0 <= len(records) <= 5000
Each record contains an integer id and a list of integer tokens
The total number of input tokens can be large
Only the first record for each id is processed
Only complete blocks are returned

Function Signature

def pack_stream(records, block_size, eos_token):

Your question is Streaming Data Pipeline Optimization. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Optimize a Python-based data pipeline to handle massive streaming inputs for model fine-tuning.

records is a list of dictionaries with integer id and integer-list tokens; block_size is a positive integer. Preserve token order and do not add an EOS token for empty or duplicate records.

Constraints

1 <= block_size
0 <= len(records) <= 5000
Each record contains an integer id and a list of integer tokens
The total number of input tokens can be large
Only the first record for each id is processed
Only complete blocks are returned

Function Signature

def pack_stream(records, block_size, eos_token):

Your question is Streaming Data Pipeline Optimization. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.