Your question is Designing PySpark ETL on Databricks. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Can you explain how you would design an ETL pipeline using PySpark and Databricks for real-time data processing?
Focus on how you would ingest streaming data, transform it with PySpark, and load it into Databricks Delta tables with low latency. Please cover schema handling, deduplication, checkpointing, and how you would monitor and recover failed runs.