Your question is Chunking for Vector Databases. Start with the requirements on the right.
Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.
Wipro ai360 needs consistently sized text segments before generating embeddings for retrieval. Given unstructured text, split it into overlapping word-based chunks while preserving the original word order.
First normalize whitespace by treating consecutive spaces, tabs, and newlines as a single separator. Then create chunks containing at most max_words tokens. Each token is a whitespace-delimited word, and punctuation remains attached to its word. Consecutive chunks must share exactly overlap words whenever enough words remain. The next chunk starts max_words - overlap positions after the current chunk starts.
Return metadata for each chunk as a dictionary with:
text: the normalized chunk textstart: inclusive starting word index in the normalized token listend: exclusive ending word indexReturn an empty list for empty or whitespace-only input. You may assume the numeric arguments satisfy the constraints.
def chunk_text(text, max_words, overlap):