Efficient Log Parsing and Validation

HardPython

Problem

Splunk ingestion pipelines must parse high-volume event lines while rejecting malformed records without stopping the batch. Each line contains space-separated key=value fields, where values may be quoted and may contain escaped quotes or backslashes.

Implement parse_logs(lines) to parse and validate every line. Return a dictionary with events, a list of valid parsed events, and invalid, the number of rejected lines.

A valid line must contain exactly these fields, with no duplicates or unknown keys:

ts: a decimal Unix timestamp from 0 through 2147483647
level: one of INFO, WARN, or ERROR
service: one or more letters, digits, underscores, or hyphens
request_id: exactly 16 hexadecimal characters
msg: a quoted string; it may contain spaces, escaped quotes (\\"), and escaped backslashes (\\\\)

Decode valid quoted values before returning them. Store ts as an integer. Preserve the other fields as strings. Lines may have leading or trailing whitespace.

Constraints

1 <= len(lines) <= 10^5
Each line contains at most 10^4 characters
The total input size is at most 10^6 characters
A valid line contains exactly five distinct fields
ts is an integer from 0 through 2147483647
request_id contains exactly 16 hexadecimal characters

Function Signature

def parse_logs(lines):

Implement parse_logs(lines) to parse and validate every line. Return a dictionary with events, a list of valid parsed events, and invalid, the number of rejected lines.

A valid line must contain exactly these fields, with no duplicates or unknown keys:

ts: a decimal Unix timestamp from 0 through 2147483647
level: one of INFO, WARN, or ERROR
service: one or more letters, digits, underscores, or hyphens
request_id: exactly 16 hexadecimal characters
msg: a quoted string; it may contain spaces, escaped quotes (\\"), and escaped backslashes (\\\\)

Decode valid quoted values before returning them. Store ts as an integer. Preserve the other fields as strings. Lines may have leading or trailing whitespace.