Explain how you would perform a diff between two large datasets using Spark to ensure scalability and avoid memory overflow.
Use the provided PostgreSQL tables as the comparison inputs. Return every record that is missing, extra, or changed between the datasets.
record_idrecord_id, diff_type, source values, and target valuesMISSING_IN_TARGET, EXTRA_IN_TARGET, and CHANGED records onlyrecord_id ascending| Column | Type | Description |
|---|---|---|
| record_idPK | BIGINT | Stable identifier used to match records |
| account_id | VARCHAR(30) | Account identifier |
| status | VARCHAR(20) | Current account status |
| balance | NUMERIC(14,2) | Reported account balance |
| as_of_date | DATE | Date associated with the record |
| Column | Type | Description |
|---|---|---|
| record_idPK | BIGINT | Stable identifier used to match records |
| account_id | VARCHAR(30) | Account identifier |
| status | VARCHAR(20) | Current account status |
| balance | NUMERIC(14,2) | Reported account balance |
| as_of_date | DATE | Date associated with the record |