Chunking Unstructured Documents | Dataford Interview Questions

Write a Python function to parse and chunk large unstructured documents efficiently for downstream vector embedding. Normalize arbitrary whitespace, preserve token order, and return overlapping chunks whose size is bounded by a supplied word limit. Define the function signature and input/output shapes below.

Constraints

0 <= len(document) <= 1000000 characters
1 <= max_words <= 100000
0 <= overlap < max_words
The document contains arbitrary Unicode text separated by whitespace
Punctuation remains attached to the surrounding whitespace-delimited token

Function Signature

def chunk_document(document, max_words, overlap):

Your question is Chunking Unstructured Documents. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Constraints

0 <= len(document) <= 1000000 characters
1 <= max_words <= 100000
0 <= overlap < max_words
The document contains arbitrary Unicode text separated by whitespace
Punctuation remains attached to the surrounding whitespace-delimited token

Function Signature

def chunk_document(document, max_words, overlap):

Your question is Chunking Unstructured Documents. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.