Design payment-event processing around durable identifiers, atomic updates and recovery. Learn what to test when a webhook arrives twice or processing stops.
An idempotent payment workflow gives one logical operation one intended effect, even when delivery or execution repeats. That is a property of the business transition, not a promise that a network message arrives once. Treat the receiving endpoint, event storage, state transition and downstream work as separate failure boundaries. A deduplication flag is useful only when its relationship to the actual effect is clear.
Choose identifiers for the right boundary
An incoming event identifier distinguishes deliveries of the same event. A payment or order identifier ties different events to one business object. An outbound request idempotency key identifies an attempted provider operation. Do not use these interchangeably. Two different events may describe changes to the same payment, while two legitimate purchases may have the same customer and amount.
Persist before acknowledging work you need to keep
Verify the incoming message using the provider's documented mechanism, then durably record the event or enqueue it before acknowledging successful receipt. Keep slow processing out of the request path where practical. If the durable write fails, the endpoint must not imply that work is safely retained. Protect event storage with suitable access and retention rules because payloads may contain personal data.
| Interruption point | Possible failure | What to prove |
|---|---|---|
| Before durable event storage | Message is acknowledged but forgotten | Acknowledgement follows successful durable acceptance. |
| After state change, before event completion | Retry repeats the financial effect | A transaction or equivalent durable guard links the effect to its identity. |
| After committing, before a downstream call | Local state is correct but delivery is missing | An outbox or reconciled job can safely resume downstream work. |
Use state transitions that tolerate late information
Define which states are allowed to follow which. Avoid updating records solely because an event arrived last. Where provider events are insufficient to establish the current state, retrieve the relevant object or reconcile against the provider's authoritative records using its documented semantics. Store the decision and identifiers that explain why an event was applied, ignored or sent for investigation.
Test the uncomfortable sequence
- Deliver the same test event sequentially and concurrently; verify one intended effect, not just one successful HTTP response.
- Stop the worker at each durable boundary and resume it. Verify both the event status and downstream result.
- Deliver a relevant sequence late or in a different order. Confirm the state model preserves valid outcomes and exposes unresolved cases.
A processing dashboard should show unfinished events and their age, not just request success rates. Keep a controlled replay procedure with an operator, audit trail and limits. The ability to replay is only useful if replaying does not bypass the same protections used by normal delivery.
- Fintech audit
- Payment reconciliation: explain every unmatched transaction
- Fintech ledger design: balances, corrections and audit trails
Frequently asked questions
Is an in-memory set of processed event IDs enough?
No for a durable financial workflow. Process restarts and multiple workers require shared persistent evidence linked to the protected operation.
Does provider request idempotency protect our webhook handler?
It protects the provider request boundary under that provider's rules. Your incoming-event processing and downstream side effects still need their own controls.
Should we store events that we do not process?
Choose an explicit policy based on relevance, investigation needs and data minimisation. Do not retain sensitive payloads indefinitely merely because storage is convenient.
Can a replay create another refund or fulfilment?
It can if the business operation lacks a durable guard. Test replay against synthetic records and verify each externally visible effect.
How do we know processing is stuck?
Track durable event status, age, retry count and reconciliation results. An endpoint returning successful responses does not prove workers completed the business operation.
Bring the scope. We will help make it buildable.
Share the user journey, integrations and launch constraints. We can clarify the scope and prepare an estimate with assumptions and exclusions.
Further reading
Payment reconciliation: explain every unmatched transaction
Build a repeatable comparison between provider records and your ledger using stable identifiers, explicit cut-offs and an owned exception queue.
Fintech ledger design: balances, corrections and audit trails
Separate financial entries, displayed balances and external settlement. Define invariants and correction paths before relying on ledger totals.
Payment integration audit: duplicate charges and missing payments
Trace a payment across checkout, provider events and internal records. Build an audit checklist that distinguishes duplicate processing from reporting delays.