The problem was that the app was fast and felt slow. Those are different problems with different fixes.

Measuring the thing that hurts

The first change was in what we tracked. We had endpoint latency, which told us the backend was healthy, and crash-free sessions, which told us the app wasn't falling over. Neither of those describes what a person experiences between opening a screen and finishing a purchase.

So we instrumented the funnel by step — screen opened, amount entered, payment method selected, provider handoff, confirmation — with timestamps at each. The picture that fell out was immediate and slightly embarrassing. Almost nobody dropped off at the steps we'd worried about. They dropped off during the provider handoff, in a window where the app showed a spinner and no other information for what could be several seconds.

People weren't leaving because it was slow. They were leaving because they didn't know if it was working, and when money is involved, uncertainty makes people back out.

Filling the silence

The fix wasn't making the provider faster — we don't control that. It was making the wait legible.

Instead of an indeterminate spinner, the handoff screen now says what's happening and roughly what to expect: contacting your bank, confirming the payment, updating your balance. Each stage advances as the actual state changes, driven by the same status polling that was already running. The total duration barely changed. The drop-off at that step fell substantially, because a progress indicator that names its steps is a promise that something is still happening.

The related change was making it safe to wait. If someone backgrounds the app mid-payment, they now come back to the same in-progress state rather than a blank buy screen, because the transaction state lives on the server and the app reconstructs from it on resume. Previously, coming back to a fresh screen made people assume the payment had failed, and some of them started it again — which was its own class of problem.

Removing taps that weren't buying anything

The other half was more ordinary. Every field on the buy screen got the same question: does the user have to make this choice, or are we asking because it was easier to ask than to decide?

Payment method defaulted to whatever they used last. Quantity got preset amounts alongside the free-text field, because most purchases cluster around a few round numbers. The confirmation step, which had been a full screen, became a sheet over the current context — so the transition costs nothing and going back doesn't feel like losing your place.

None of these are clever. Together they took a meaningful number of interactions out of the common path, and the common path is where almost everyone lives.

What I'd do differently

I'd instrument the funnel before touching anything. We spent time optimising a list render and a couple of API calls at the start of this work, on the reasonable-sounding theory that faster is better. It was faster. It changed nothing, because those weren't the moments where people were giving up.

The general version: performance work has two halves, and engineers are much better equipped for the first one. Making the code fast is measurable, satisfying, and has good tooling. Making the product feel fast is about what someone knows at each moment — whether they can tell it's working, whether they can tell how long is left, whether they believe their money is safe. A profiler will never show you that. Step-level funnel data will show you exactly where it hurts, and it's about an afternoon of work to add.

It wasn't the provider's problem.

Two mouths, one story

Our payment flow had two independent ways of learning that a transaction had succeeded. The provider sent us a webhook. The mobile client, which had been sitting on a spinner, also polled a status endpoint and told us what it found. Both paths eventually called the same internal function to credit the user's gold balance and mark the order complete.

For almost every transaction, one of them won by a comfortable margin and the second arrived to find the order already in a terminal state. But "already in a terminal state" was a read, and the write that followed it was a separate statement. Two requests that both read pending will both proceed to write, and no amount of careful ordering inside the handler fixes that, because the handlers were running in different processes.

// Roughly what we had. The check and the write are not atomic.
const order = await db.orders.findById(orderId)
if (order.status === 'completed') return
await creditGoldBalance(order.userId, order.grams)
await db.orders.update(orderId, { status: 'completed' })

The window is small. On a good day it is a few milliseconds wide. It is also open on every single transaction, which is why "small window" is not a defence — you are just waiting for enough volume to walk through it.

Making the write decide

The fix that actually holds is to stop asking the database a question and start making it enforce an answer. Every credit operation got an idempotency key derived from the provider's transaction reference — not from anything we generated, because our own request IDs differ between the webhook path and the polling path, which is exactly the thing that let the duplicate through.

create unique index credits_idempotency_key_uniq
  on credits (idempotency_key);

Then the credit becomes an insert that is allowed to fail:

try {
  await db.credits.insert({
    idempotencyKey: providerTxnRef,
    userId,
    grams,
  })
} catch (err) {
  if (err.code === UNIQUE_VIOLATION) return // someone else got here first
  throw err
}

Now it doesn't matter who arrives first or how many arrive at once. The database picks a winner and everyone else quietly discovers they lost. This is the whole trick: push the decision down to the one component that can make it atomically, instead of trying to coordinate it upward.

One source of truth

The second half of the fix was deciding who is allowed to have an opinion. The client polling path was never meant to be authoritative — it existed so the app could update its UI. Somewhere along the way it had acquired the power to mutate balances, because that was the convenient place to put it when the feature was first written.

So we cut it. The client now polls a read-only endpoint that reports what the backend believes. The backend forms that belief from the provider's webhook and from a reconciliation job that periodically fetches recent transactions from the provider and compares them against our own records. If the provider says a payment succeeded and we have no credit for it, that's a discrepancy the job repairs and flags. If we have a credit the provider doesn't recognise, that's an alert someone reads.

The reconciliation job is the part I'd argue for hardest to anyone building this. Webhooks get dropped. Providers have outages, and when they come back they don't always replay. A periodic sweep that treats the provider as the source of truth for what happened and our database as the source of truth for what we did about it will catch the class of failure that no amount of careful webhook handling covers.

Proving it

The thing I'd been missing before this bug was a way to actually test concurrent delivery. Unit tests call the handler once. The bug only exists when it's called twice at the same time.

What worked was capturing real webhook payloads from staging and building a small harness that replayed them — the same event, several times, concurrently, with the polling path firing alongside. Before the fix it reproduced the double credit reliably within a few runs. After the fix it didn't, and more usefully, it kept not reproducing it when we later refactored the payment service. That harness is now part of the deploy checks, which means the test that found the bug is the test that stops it coming back.

What I took from it

The general lesson isn't "use idempotency keys" — that's the specific lesson. The general one is that when two systems can both tell you the same fact, you have to decide in advance which one you believe, and enforce that decision somewhere it can't be bypassed. We had never made that decision. We'd just built two paths at two different times, each one sensible on its own, and the contradiction between them sat there quietly until volume found it.

Money makes this visible fast. But the same shape shows up anywhere you have a webhook and a poll, a cache and an origin, or an optimistic client update and a server confirmation. Somebody has to be right. Pick who, and then make the schema hold you to it.

That cold-start problem is what pushed me toward retrieval-augmented generation. What I learned building it is that the interesting work was never in the generation step.

The shape of the thing

The pipeline is unremarkable. Each product gets an embedding built from its title, description, region, and category. A shopper viewing a product triggers a nearest-neighbour search over that space, filtered by what's actually in stock. The top handful of results go to a model along with the current product, and it writes a short line explaining why these go together.

const neighbours = await db.execute(sql`
  select id, title, region, embedding <=> ${queryEmbedding} as distance
  from products
  where in_stock = true and id != ${currentProductId}
  order by distance
  limit 8
`)

The first version of this was bad in a way that took me a while to see. The recommendations were plausible — that's the problem with plausible — and the copy the model wrote around them was fluent and confident. It read well. It just wasn't recommending things anyone wanted.

The retrieval was the product

What fixed it was almost entirely upstream of the model.

What goes into the embedding matters more than which embedding model you use. My first pass embedded the full product description, which meant long descriptions dominated the space and every verbose listing looked similar to every other verbose listing. Cutting it down to title, category, region, and a short curated blurb produced dramatically better neighbours. Same model, different input.

Filters belong in the query, not after it. I initially fetched the top 8 neighbours and then dropped the out-of-stock ones, which sometimes left two. Pushing the stock filter into the vector query means you always get eight real candidates. Obvious in hindsight; easy to get wrong when you're treating the vector store as a black box that returns "the answer".

Distance needs a floor. Nearest neighbour always returns something. If nothing in the catalogue is genuinely close, it returns the least-far thing, and the model will cheerfully write a paragraph about why a pickle jar pairs with a wool shawl. A distance threshold, below which we show nothing rather than something, made the recommendations trustworthy — and trustworthiness is what makes people click the second one.

That last point is the one I'd underline. The temptation with a generative layer is to always produce output, because output feels like value. Showing nothing is a valid result and it protects every other recommendation you do show.

Where the model actually helped

Having said all that, the generation step wasn't decorative. Once retrieval was solid, a one-line explanation of the connection — "from the same valley as the tea you're looking at" — measurably outperformed showing the same products with no copy at all. People need a reason, and writing a reason for every pair in a catalogue by hand isn't feasible.

But it only works on top of good retrieval. Good copy about bad recommendations is worse than no copy, because it spends the credibility you'd otherwise have.

Keeping it inspectable

The thing I built early and never regretted was an internal page that shows, for any product, the retrieved neighbours and their raw distances before any of it reaches a model. When a recommendation looks wrong, I can tell in about ten seconds whether the retrieval was wrong or the copy was wrong. Those have completely different fixes, and without that page you're guessing between them.

If you build one of these, build that page first. It's forty lines and it's the difference between debugging a system and staring at it.

The 40%

Conversions went up about 40% after this shipped. I want to be careful about what that number means: it's a real measurement against the prior baseline, but it's a small storefront, and the recommendation layer went in alongside other work. The honest version is that surfacing relevant products where there were previously none moved the number a lot, and that most of the moving was done by the retrieval quality rather than the language model on top of it.

Which is, I think, the useful takeaway. RAG is a retrieval system with a writing assistant attached. If the retrieval is good, the assistant makes it better. If the retrieval is bad, the assistant makes it convincing, and that's a considerably worse outcome than doing nothing.