Parse 10GB Log and Count PCIe Errors | Dataford Interview Questions

An AMD EPYC system can produce log files too large to load into memory at once. Write a function that streams log lines and counts occurrences of selected PCIe error codes.

Each relevant log entry contains the exact pattern PCIe_ERROR code=CODE, where CODE consists only of uppercase letters, digits, underscores, and hyphens. Ignore lines without this pattern and ignore codes that are not in the requested set. A line contains at most one PCIe error entry.

Formal Specification

Implement count_pcie_errors(lines, error_codes).

lines is an iterable of strings, such as a file object. The function must process it incrementally and must not convert the complete input into a list.
error_codes is a list of distinct strings identifying codes to count.
Return a dictionary mapping every requested code to its frequency, including codes with frequency zero.

Constraints

1 <= len(error_codes) <= 10^4
The total log size can be approximately 10 GB
Each line contains at most 10^4 characters
Error codes are case-sensitive
Input lines may contain malformed or unrelated text
Each line contains at most one PCIe error entry

Function Signature

def count_pcie_errors(lines, error_codes):

Your question is Parse 10GB Log and Count PCIe Errors. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem