Context

Apache Spark performance and correctness often depend on understanding when work is planned vs when it is executed. Lazy evaluation is the mechanism that lets Spark build an execution plan before running it.

Question

Explain the concept of lazy evaluation in Apache Spark (RDD/DataFrame/Dataset APIs).

In your answer, cover:

1. What lazy evaluation means in Spark and why Spark uses it.
2. The difference between transformations and actions, and how they relate to triggering execution.
3. How Spark builds a lineage/DAG, what a job/stage/task is at a high level, and how optimizations (e.g., predicate pushdown / query planning) benefit from laziness.

Scope Guidance

Assume the interviewer expects a clear mental model (DAG + trigger) and practical implications (debugging, performance, caching/checkpointing), not deep internals of Catalyst or Tungsten.