Real-Time Data Quality Monitoring | Dataford Interview Questions

How do you enforce data quality rules and monitoring in real-time streaming architectures?

Write a PostgreSQL query using ingestion_batches and streaming_events to evaluate each ingestion batch against the stated validation rules. Identify batches with invalid records or an event-count mismatch.

Output

One row per ingestion batch, including batches with no events
Columns: batch_id, source_name, received_event_count, expected_event_count, violating_event_count, and quality_status
Sort by received_at ascending, then batch_id ascending

Schema

ingestion_batches

Column	Type	Description
batch_idPK	INT	Unique ingestion batch identifier
source_name	VARCHAR(50)	Streaming source name
received_at	TIMESTAMP	Time the batch was received
expected_event_count	INT	Expected number of events in the batch

streaming_events

Column	Type	Description
record_idPK	INT	Unique stored event record identifier
batch_id	INT	Associated ingestion batch
event_id	VARCHAR(50)	Source event identifier
event_type	VARCHAR(40)	Type of streaming event
event_ts	TIMESTAMP	Timestamp supplied by the event producer
amount	NUMERIC(12,2)	Event monetary amount

Your question is Real-Time Data Quality Monitoring. Start with the requirements and the two tables in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem