Real-Time Data Processing

If we get a data stream which consists of some predefined types of data (say logs, metrics, metaData etc.), how will we process it in real time and extract?

Asked in the HM round stage.

Explain a practical end-to-end design for identifying record types, validating and routing events, extracting useful fields, and applying ML inference where appropriate. Discuss how your design handles malformed data, late or duplicate events, schema evolution, backpressure, and operational failures.

1. Define the ingestion and stream-processing architecture.
2. Explain parsing, normalization, validation, and type-specific extraction.
3. Describe online inference, state management, and output storage.
4. Cover offline training, feedback, monitoring, and model updates.
5. Identify important failure modes and recovery strategies.