Context

In distributed data processing, changing the number of partitions affects parallelism, shuffle cost, and skew. Spark exposes two APIs—repartition() and coalesce()—that both change partitioning but with different guarantees and costs.

Core Question

Explain the difference between repartition() and coalesce() in Spark.

Address these points:

1. What each operation does to the number of partitions and the distribution of records.
2. Whether it triggers a full shuffle, and what that implies for network and disk I/O.
3. When you would prefer one over the other (e.g., increasing partitions, decreasing partitions, mitigating skew, preparing for joins/writes).

Scope Guidance

Assume the interviewer expects practical engineering depth: mention shuffle boundaries, narrow vs wide dependencies, and typical performance implications. You do not need to write Spark code, but you should be able to reason about execution plans and trade-offs.