Word Count in Large File | Dataford Interview Questions

Given 1TB of a file, how to check the word count using any language?

Asked in the Panel interview stage. Implement a streaming word counter that processes successive text chunks and never loads the complete file into memory.

Input and Output

chunks is an iterable of strings representing consecutive file chunks. Return the number of whitespace-delimited words. A word may span multiple chunks, and any Unicode whitespace separates words.

Constraints

The total file size may be as large as 1TB.
chunks is an iterable of strings in file order.
A word is a maximal sequence of non-whitespace characters.
Chunks may split a word at any character boundary.
Unicode whitespace is treated as a separator.

Function Signature

def count_words(chunks):

Your question is Word Count in Large File. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem