The duplicate charge that taught me idempotency

The report came in the way these always do: not from monitoring, but from a support ticket. A customer had bought gold once and been charged twice. Then another. Not many — a handful over a few days — which is somehow worse than a lot, because a handful is easy to explain away as the provider's problem.

It wasn't the provider's problem.

Two mouths, one story

Our payment flow had two independent ways of learning that a transaction had succeeded. The provider sent us a webhook. The mobile client, which had been sitting on a spinner, also polled a status endpoint and told us what it found. Both paths eventually called the same internal function to credit the user's gold balance and mark the order complete.

For almost every transaction, one of them won by a comfortable margin and the second arrived to find the order already in a terminal state. But "already in a terminal state" was a read, and the write that followed it was a separate statement. Two requests that both read pending will both proceed to write, and no amount of careful ordering inside the handler fixes that, because the handlers were running in different processes.

// Roughly what we had. The check and the write are not atomic.
const order = await db.orders.findById(orderId)
if (order.status === 'completed') return
await creditGoldBalance(order.userId, order.grams)
await db.orders.update(orderId, { status: 'completed' })

The window is small. On a good day it is a few milliseconds wide. It is also open on every single transaction, which is why "small window" is not a defence — you are just waiting for enough volume to walk through it.

Making the write decide

The fix that actually holds is to stop asking the database a question and start making it enforce an answer. Every credit operation got an idempotency key derived from the provider's transaction reference — not from anything we generated, because our own request IDs differ between the webhook path and the polling path, which is exactly the thing that let the duplicate through.

create unique index credits_idempotency_key_uniq
  on credits (idempotency_key);

Then the credit becomes an insert that is allowed to fail:

try {
  await db.credits.insert({
    idempotencyKey: providerTxnRef,
    userId,
    grams,
  })
} catch (err) {
  if (err.code === UNIQUE_VIOLATION) return // someone else got here first
  throw err
}

Now it doesn't matter who arrives first or how many arrive at once. The database picks a winner and everyone else quietly discovers they lost. This is the whole trick: push the decision down to the one component that can make it atomically, instead of trying to coordinate it upward.

One source of truth

The second half of the fix was deciding who is allowed to have an opinion. The client polling path was never meant to be authoritative — it existed so the app could update its UI. Somewhere along the way it had acquired the power to mutate balances, because that was the convenient place to put it when the feature was first written.

So we cut it. The client now polls a read-only endpoint that reports what the backend believes. The backend forms that belief from the provider's webhook and from a reconciliation job that periodically fetches recent transactions from the provider and compares them against our own records. If the provider says a payment succeeded and we have no credit for it, that's a discrepancy the job repairs and flags. If we have a credit the provider doesn't recognise, that's an alert someone reads.

The reconciliation job is the part I'd argue for hardest to anyone building this. Webhooks get dropped. Providers have outages, and when they come back they don't always replay. A periodic sweep that treats the provider as the source of truth for what happened and our database as the source of truth for what we did about it will catch the class of failure that no amount of careful webhook handling covers.

Proving it

The thing I'd been missing before this bug was a way to actually test concurrent delivery. Unit tests call the handler once. The bug only exists when it's called twice at the same time.

What worked was capturing real webhook payloads from staging and building a small harness that replayed them — the same event, several times, concurrently, with the polling path firing alongside. Before the fix it reproduced the double credit reliably within a few runs. After the fix it didn't, and more usefully, it kept not reproducing it when we later refactored the payment service. That harness is now part of the deploy checks, which means the test that found the bug is the test that stops it coming back.

What I took from it

The general lesson isn't "use idempotency keys" — that's the specific lesson. The general one is that when two systems can both tell you the same fact, you have to decide in advance which one you believe, and enforce that decision somewhere it can't be bypassed. We had never made that decision. We'd just built two paths at two different times, each one sensible on its own, and the contradiction between them sat there quietly until volume found it.

Money makes this visible fast. But the same shape shows up anywhere you have a webhook and a poll, a cache and an origin, or an optimistic client update and a server confirmation. Somebody has to be right. Pick who, and then make the schema hold you to it.