Diff Between Large Datasets with Spark

Explain how you would perform a diff between two large datasets using Spark to ensure scalability and avoid memory overflow.

Use the provided PostgreSQL tables as the comparison inputs. Return every record that is missing, extra, or changed between the datasets.

Output

1. One row per differing record_id
2. Columns: record_id, diff_type, source values, and target values
3. Include MISSING_IN_TARGET, EXTRA_IN_TARGET, and CHANGED records only
4. Order by record_id ascending

Output details

• Return columns exactly in this order: record_id, diff_type, source_account_id, source_status, source_balance, source_as_of_date, target_account_id, target_status, target_balance, target_as_of_date.
• Compare account_id, status, balance, and as_of_date values for each record_id.
• Treat a NULL versus non-NULL value as changed, but treat two NULL values as equal.