Build Reliable Dashboard Refresh Pipeline

Scenario

You are responsible for the data pipeline that powers customer-facing operational dashboards for a utility analytics product. The current workflow refreshes visualizations after upstream meter, outage, and billing data lands, but users regularly see mismatched totals between dashboards and exported reports after reruns or late-arriving corrections. An internal audit has flagged the process as non-repeatable because the same reporting date can produce different results depending on job timing and partial failures. You need to redesign the workflow so dashboard updates are reliable, repeatable, and easy to recover without manual intervention.

Current State

Question

How would you redesign this pipeline so each dashboard refresh is deterministic, idempotent, and auditable even when upstream files arrive late, are corrected, or need to be replayed? Explain the end-to-end workflow you would build from ingestion through transformation, validation, publication, and rollback for the visualization layer.