SQL, PySpark, and Azure Stack | Dataford Interview Questions

Tell me about your SQL and PySpark skills and how you use Azure services in your data engineering work at scale.

For the SQL portion, summarize January and February 2024 Azure Synapse pipeline runs by workspace and month using the supplied metadata. Include only workspace-month combinations with at least two runs.

Output

One row per workspace and month with workspace_name, report_month, total_runs, successful_runs, success_rate_pct, and latest_status
Include the latest run status and order by workspace_name, then report_month ascending

Schema

synapse_workspaces

Column	Type	Description
workspace_idPK	INT	Unique Azure Synapse workspace identifier
workspace_name	VARCHAR(100)	Readable workspace name
environment	VARCHAR(20)	Workspace environment classification

synapse_pipeline_runs

Column	Type	Description
run_idPK	INT	Unique pipeline run identifier
workspace_id	INT	Workspace associated with the run
pipeline_name	VARCHAR(100)	Executed pipeline name
status	VARCHAR(20)	Pipeline outcome
started_at	TIMESTAMP	Pipeline start timestamp

Your question is SQL, PySpark, and Azure Stack. Start with the requirements and the two tables in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem