Whenever you're ready, walk me through how you'd approach this.
Can you explain Spark architecture?
Describe the roles of the driver, cluster manager, executors, jobs, stages, and tasks. Explain how Spark builds and executes a DAG, handles transformations and actions, performs shuffles, and provides fault tolerance. For a practical answer, include how you would structure a batch or Structured Streaming application, configure resources, monitor execution, and diagnose performance issues such as data skew, excessive shuffles, executor failures, and insufficient parallelism.