Design Idempotent Payment Event Pipeline

Scenario

You are redesigning the payment processing pipeline for a digital banking platform after an audit found duplicate captures and inconsistent downstream ledger records during retries, consumer restarts, and backfills. The platform is moving from synchronous request-driven writes to an event-driven architecture so payment authorization, capture, settlement, and reconciliation can be processed independently. The main pain point is that the same business operation may be delivered multiple times by API clients, brokers, and batch replay jobs, but it must produce exactly one financial effect. You need a design that keeps operational systems and analytical stores consistent without blocking throughput.

Current State

Question

How would you design an idempotency framework across the API, streaming pipeline, storage layers, and replay workflows so duplicate requests and duplicate events never create duplicate financial side effects, while still supporting retries, late events, and backfills at this scale?