Whenever you're ready, walk me through how you'd approach this.
How would you optimize a data pipeline for processing large geospatial datasets?
Discuss a practical design covering ingestion, spatial partitioning, distributed transformations, storage formats, batch and streaming workloads, and orchestration. Explain how you would handle coordinate-system normalization, spatial joins, skewed regions, late or duplicate records, incremental loads, backfills, and data-quality validation. Include concrete technology choices, representative implementation code, monitoring metrics, failure recovery, and trade-offs between query performance, storage cost, and processing latency.