Build Billing Sync Reconciliation Pipeline

Context

Anrok ingests transactions, invoices, credit notes, and subscription updates from third-party billing platforms into the Anrok tax platform. Today, some integrations rely on a mix of scheduled API pulls and webhook handlers, which leads to duplicate events, missed updates during provider outages, and inconsistent tax state when upstream systems are eventually consistent.

Design a resilient pipeline that keeps Anrok's billing ledger and downstream tax calculation surfaces accurate despite API rate limits, out-of-order webhooks, retries, and delayed updates from providers such as Stripe or Chargebee.

Scale Requirements

• Connected merchants: 8,000
• Providers: 6 billing systems, each with different API quotas and webhook semantics
• Peak webhook volume: 15,000 events/minute
• Backfill volume: up to 250M billing objects across 24 months
• Latency target: webhook to Anrok queryable state in < 2 minutes P95
• Reconciliation SLA: provider state converges in Anrok within 30 minutes for 99.9% of objects
• Storage: 15 TB raw history, 3-year retention for auditability

Requirements

1. Ingest provider webhooks into Anrok with signature verification, replay protection, and durable storage.
2. Design pull-based sync jobs that respect per-provider and per-merchant rate limits while supporting incremental sync and historical backfills.
3. Handle eventual consistency: a webhook may arrive before the provider API reflects the latest object state.
4. Guarantee idempotent processing for duplicate webhooks, retried API pages, and re-run backfills.
5. Model raw events, canonical billing objects, and reconciliation status so downstream tax calculations read a consistent view.
6. Define orchestration for near-real-time processing, scheduled reconciliation, and targeted reprocessing.
7. Include data quality checks, observability, and on-call alerting.

Constraints

• AWS-first stack; prefer managed services where possible.
• Team of 5 engineers; operational simplicity matters more than perfect theoretical throughput.
• Must support SOC 2 auditability and immutable raw event retention.
• Budget target: <$35K/month incremental infrastructure cost.
• Some providers expose only cursor-based pagination and strict burst limits (for example, 100 requests/sec/account).