Tell me about a time you optimized a Spark query and used SQL and Python to solve a data problem end to end.
Use the provided query-run and stage-metric data to demonstrate the SQL portion of that workflow. Return the latest completed run for each query, compare it with the previous completed run, and identify the slowest stage.
query_id, owner, run_id, total_duration_seconds, prior_duration_seconds, duration_delta_pct, bottleneck_stage, optimization_flagduration_delta_pct descending, with nulls last, then query_id ascending| Column | Type | Description |
|---|---|---|
| query_idPK | INT | Logical identifier for a Spark query |
| owner | VARCHAR(100) | Query owner |
| query_text | TEXT | SQL text submitted to Spark |
| Column | Type | Description |
|---|---|---|
| run_idPK | INT | Unique execution identifier |
| query_id | INT | Referenced logical query |
| run_started_at | TIMESTAMP | Execution start time |
| status | VARCHAR(20) | Execution status |
| input_gb | NUMERIC(10,2) | Input data volume in gigabytes |
| Column | Type | Description |
|---|---|---|
| run_id | INT | Referenced query execution |
| stage_id | INT | Stage identifier within a run |
| stage_name | VARCHAR(100) | Human-readable stage name |
| duration_seconds | NUMERIC(12,2) | Stage execution duration |
| shuffle_read_gb | NUMERIC(10,2) | Shuffle read volume in gigabytes |
| spill_gb | NUMERIC(10,2) | Spill volume in gigabytes |