Context

Apache Spark runs distributed computations on clusters where executors and machines can fail. A strong answer explains how Spark recovers results without requiring full data replication.

Core Question

Explain how Spark handles fault tolerance in batch and streaming workloads.

Address the following:

1. RDD/DataFrame lineage: What is lineage, how is it represented (DAG), and how does it enable recomputation of lost partitions?
2. Shuffle fault tolerance: What happens when shuffle outputs are lost, and how do map/reduce stages get recomputed?
3. Checkpointing and caching: When does Spark use checkpointing, what problem does it solve (long lineage), and how does it differ from caching/persisting?

Scope Guidance

Assume the interviewer expects a systems-level explanation (driver, executors, tasks, stages) and the trade-offs (recompute vs. storage/replication). Mention at least one failure scenario (executor loss) and walk through recovery steps at a high level.