You are supporting a production eligibility pipeline, and a client reports that expected member eligibility records are missing. The source file was delivered on time, so the issue could be anywhere between file arrival, ingestion, transformation, validation, and downstream publication. You need to trace the pipeline end to end and isolate where records were dropped, delayed, or filtered out.
A client is missing eligibility records in production, but the source file was delivered on time. How would you debug the issue end to end?
File arrival metadata, checksum, and naming conventionAirflow DAG run state and upstream dependency completionSnowflake raw load counts, rejected rows, and COPY historydbt model filters, joins, and incremental predicatesQuality test failures and downstream publish status