Explain Spark internals and how it scales with the volume of data.
Using the supplied Spark event-log summary tables, write a PostgreSQL query that summarizes successful jobs with input data, stage execution, shuffle activity, and executor utilization.
job_id, app_name, input_gb, stage_count, total_stage_seconds, total_shuffle_read_gb, total_shuffle_write_gb, max_cpu_utilization_pct, input_delta_gbstarted_at, then job_id| Column | Type | Description |
|---|---|---|
| job_idPK | INT | Spark job identifier |
| app_name | VARCHAR(100) | Application name |
| started_at | TIMESTAMP | Job start timestamp |
| input_gb | DECIMAL(10,2) | Input data volume in gigabytes |
| output_gb | DECIMAL(10,2) | Output data volume in gigabytes |
| status | VARCHAR(20) | Final job status |
| Column | Type | Description |
|---|---|---|
| stage_idPK | INT | Spark stage identifier |
| job_id | INT | Owning Spark job |
| stage_name | VARCHAR(100) | Stage description |
| duration_seconds | INT | Stage duration in seconds |
| shuffle_read_gb | DECIMAL(10,2) | Shuffle data read in gigabytes |
| shuffle_write_gb | DECIMAL(10,2) | Shuffle data written in gigabytes |
| task_count | INT | Number of tasks in the stage |
| failed_tasks | INT | Number of failed tasks |
| Column | Type | Description |
|---|---|---|
| metric_idPK | INT | Executor metric record identifier |
| job_id | INT | Owning Spark job |
| executor_id | VARCHAR(30) | Executor identifier |
| peak_memory_gb | DECIMAL(10,2) | Peak executor memory usage |
| cpu_utilization_pct | DECIMAL(5,2) | Peak CPU utilization percentage |
| tasks_completed | INT | Tasks completed by the executor |