Track Data Lineage Framework | Dataford Interview Questions

What framework would you implement to track data lineage across distributed pipelines?

Write a PostgreSQL query over the supplied dataset, pipeline, and lineage-edge tables. For target dataset ID 10, return every reachable upstream dataset while preventing cycles during traversal.

Output

One row per reachable upstream dataset, excluding target ID 10
Columns: dataset_id, dataset_name, minimum_hops, lineage_path_count, pipeline_count, and pipeline_names
Sort by minimum_hops, then dataset_id

Schema

datasets

Column	Type	Description
dataset_idPK	INT	Unique dataset identifier
dataset_name	VARCHAR(100)	Dataset name
domain	VARCHAR(50)	Owning data domain

pipelines

Column	Type	Description
pipeline_idPK	INT	Unique pipeline identifier
pipeline_name	VARCHAR(100)	Pipeline name
orchestrator	VARCHAR(50)	Pipeline orchestration system

lineage_edges

Column	Type	Description
edge_idPK	INT	Unique lineage edge identifier
source_dataset_id	INT	Upstream dataset
target_dataset_id	INT	Downstream dataset
pipeline_id	INT	Pipeline creating the dependency
edge_note	VARCHAR(120)	Optional lineage metadata

Your question is Track Data Lineage Framework. Start with the requirements and the three tables in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem