What RAG actually earned us in eCommerce

Pahadi Street sells regional products from the hills — food, crafts, things with a story attached. It's the sort of catalogue where a shopper who lands on one item genuinely might want three others, and where the connection between them isn't something a "customers also bought" table can find, because there isn't enough traffic yet to populate one.

That cold-start problem is what pushed me toward retrieval-augmented generation. What I learned building it is that the interesting work was never in the generation step.

The shape of the thing

The pipeline is unremarkable. Each product gets an embedding built from its title, description, region, and category. A shopper viewing a product triggers a nearest-neighbour search over that space, filtered by what's actually in stock. The top handful of results go to a model along with the current product, and it writes a short line explaining why these go together.

const neighbours = await db.execute(sql`
  select id, title, region, embedding <=> ${queryEmbedding} as distance
  from products
  where in_stock = true and id != ${currentProductId}
  order by distance
  limit 8
`)

The first version of this was bad in a way that took me a while to see. The recommendations were plausible — that's the problem with plausible — and the copy the model wrote around them was fluent and confident. It read well. It just wasn't recommending things anyone wanted.

The retrieval was the product

What fixed it was almost entirely upstream of the model.

What goes into the embedding matters more than which embedding model you use. My first pass embedded the full product description, which meant long descriptions dominated the space and every verbose listing looked similar to every other verbose listing. Cutting it down to title, category, region, and a short curated blurb produced dramatically better neighbours. Same model, different input.

Filters belong in the query, not after it. I initially fetched the top 8 neighbours and then dropped the out-of-stock ones, which sometimes left two. Pushing the stock filter into the vector query means you always get eight real candidates. Obvious in hindsight; easy to get wrong when you're treating the vector store as a black box that returns "the answer".

Distance needs a floor. Nearest neighbour always returns something. If nothing in the catalogue is genuinely close, it returns the least-far thing, and the model will cheerfully write a paragraph about why a pickle jar pairs with a wool shawl. A distance threshold, below which we show nothing rather than something, made the recommendations trustworthy — and trustworthiness is what makes people click the second one.

That last point is the one I'd underline. The temptation with a generative layer is to always produce output, because output feels like value. Showing nothing is a valid result and it protects every other recommendation you do show.

Where the model actually helped

Having said all that, the generation step wasn't decorative. Once retrieval was solid, a one-line explanation of the connection — "from the same valley as the tea you're looking at" — measurably outperformed showing the same products with no copy at all. People need a reason, and writing a reason for every pair in a catalogue by hand isn't feasible.

But it only works on top of good retrieval. Good copy about bad recommendations is worse than no copy, because it spends the credibility you'd otherwise have.

Keeping it inspectable

The thing I built early and never regretted was an internal page that shows, for any product, the retrieved neighbours and their raw distances before any of it reaches a model. When a recommendation looks wrong, I can tell in about ten seconds whether the retrieval was wrong or the copy was wrong. Those have completely different fixes, and without that page you're guessing between them.

If you build one of these, build that page first. It's forty lines and it's the difference between debugging a system and staring at it.

The 40%

Conversions went up about 40% after this shipped. I want to be careful about what that number means: it's a real measurement against the prior baseline, but it's a small storefront, and the recommendation layer went in alongside other work. The honest version is that surfacing relevant products where there were previously none moved the number a lot, and that most of the moving was done by the retrieval quality rather than the language model on top of it.

Which is, I think, the useful takeaway. RAG is a retrieval system with a writing assistant attached. If the retrieval is good, the assistant makes it better. If the retrieval is bad, the assistant makes it convincing, and that's a considerably worse outcome than doing nothing.