There's a moment in every demo of an internal project I've run where something mildly wrong slides past everyone on the call except me. We were showing a retrieval pipeline to a Fortune 500 hospitality brand. The model generated a perfectly fluent paragraph of property copy somewhere in the Gulf, and two sentences in, it used the word "luxurious" three times. The client's actual style guide, which we had ingested, embedded, chunked, and lovingly indexed into a vector DB, said in plain English, "Avoid the word 'luxurious'. It means nothing." We had retrieved the rule and we had cited the rule, and the model, in its infinite confidence, had simply chosen to ignore it.
That, in fairness, is the polite version of what RAG does most of the time. It looks things up, and then it writes whatever it wanted to write anyway.
The hype slide goes the other way. "RAG is Retrieval-Augmented Generation, a technique that combines large language models with external knowledge sources for grounded, accurate output free of hallucination." Sure, fine. Lewis et al. wrote the original paper back in 2020 when GPT-3 was new and everyone was scared their model didn't know things (iykyk, it was a different era of being scared). Six years later the models know plenty of things, and what people get wrong is thinking the hard part is retrieval. Retrieval is the easy part. There are a hundred libraries that will chunk a PDF, embed it, and dump it into Pinecone or pgvector by Wednesday. The hard parts are two things, what you look up, and what you do with it once you have it.
arxiv · 2005.114012020Lewis, Perez, Piktus, et al.The OG. Coined the acronym, framed retrieval as a non-parametric memory. Skimmable in 25 minutes.→ open sourceWhen the corpus is a knowledge base of factual documents (think a help center, a policy manual, a product spec), RAG has a fighting chance. The questions have answers, the answers exist in the corpus, and you can grade success by checking whether the cited document actually contains the claim. Brand voice corpora are different beasts. Style is implicit, and nobody writes a sentence that says "we, the brand, hereby use the Oxford comma and prefer verbs of two syllables over three." Style is what's left over after you read 200 issues of the magazine and absorb it through your skin.
I edited UNaB Magazine for seven years. We had about sixty freelance writers at any given time, and the way new writers learned the house voice was not by reading our style document, all twelve pages of it. It was by getting their first three submissions back covered in red ink and slowly realising what we did and didn't tolerate. We rejected the word "plethora" on sight, allowed sentence fragments, and forbade exclamation marks except in dialogue. None of that was in the stylebook. The stylebook said "be clear and warm," which was useless.
Every generation is, in effect, a brand new freelancer walking in for their first day on the job and being asked to write a homepage for a brand they've never read more than fragments of.
Let's follow a single query through the pipeline. A marketer at the client side types, "Write me a product description of sixty words for our new sustainable carry-on, in our voice, for the US homepage," and here's what happens.
Five things break, in roughly that order, in roughly every project. The default LangChain text splitter (and yes I have a rant about this, half written, saved as a draft on my Substack) will happily cut sentences in half and embed the halves, and the halves embed badly and retrieve badly. You search for "return policy" and get the second half of a sentence that started "Unlike our return policy," which means the chunk is about something that is not, technically, the return policy. Embedding model choice is the next failure point. Everyone reaches for OpenAI's text-embedding-3-large because it's a known quantity. It's fine. It is also not optimised for your domain. For one client whose corpus was largely in Bangla, we found a multilingual model from BAAI outperformed it on retrieval recall by enough margin to matter, the difference between a pipeline that ships and one that gets quietly killed in QBR.
Then there's recall versus precision, the eternal tradeoff. Retrieve more chunks and you catch the right answer, but you also drown the model in noise. Retrieve fewer and you miss things. The naive instinct is to crank top-k up to 30 and call it a day. But Liu et al.'s "Lost in the Middle" paper from 2023 showed, with the kind of clean experimental design that most RAG papers conspicuously lack, that models built for long context systematically ignore the middle of their context window. So the chunk you retrieved at rank 14, that contains the actual answer, might as well not exist. The model's attention is on positions 1, 2, and N. The middle is where chunks go to die.
arxiv · 2307.031722023Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, LiangThe attention curve shaped like a U. Goes hard on how badly the middle of the context degrades. Section 4 is the money section.→ open sourceYou'd think the fix is "put the most important chunk first and last." You'd be right, and it's depressing how much that helps. We sandwich the chunk with the highest rerank score at position 1 and reprise it at position N. Crude, effective, and a reminder that sometimes the answer to a hard ML problem is to shrug and rearrange the list.
The fifth and last failure point took me about two years to take seriously. The prompt that decides what to do with context. RAG papers focus on retrieval quality, but real systems live or die by what the prompt says about retrieved content. "Use these documents to answer" is a different system from "Use these documents as voice examples, do not copy phrasing, and reject the generation if you cannot ground every claim in a citation." The second one makes the model nervous, in a useful way.
blog · anthropic.com/news2024Anthropic Applied AI teamContextual retrieval. The TLDR: prepend a summary, one paragraph long, of each chunk's role in the parent doc before embedding. Reduces failure rate by 49%.→ open sourceIt also happens to be the only thing that separates a working product from a Notion demo. Chris Albon has been saying "evals are all you need" for a couple of years now, and every team I know that ignored him has had to come back and build evals later, under deadline, with worse data and an angrier account director. He says it in approximately one short post a week. He is, as far as I can tell, just correct.
personongoingChris Albon (Director of ML at Wikimedia)The 'evals are all you need' refrain. Worth following purely for the consistency of the message.→ open sourceOur eval stack for brand voice RAG has four layers.
- Factual grounding. Every claim in the output must trace to a retrieved chunk. We do this with an LLM acting as judge that gets the output and the chunks and is asked, claim by claim, "is this supported." If it isn't, the generation gets flagged for review.
- Golden answer comparison. A small set, fifty maybe a hundred, of curated query and answer pairs that a senior brand strategist signed off on. Every release is regressed against the goldens before it goes anywhere near a customer surface.
- Brand fidelity score. A separate judge model, trained or prompted on the brand's own published corpus, scores every output on a 0 to 1 scale. The first time we ran it on a hospitality client's homepage copy, the score was humbling. The next three rounds of prompt and chunking work only got us to "the brand's distant relative who has read most of its mail." That gap is what convinced me to stop calling demos ready for production.
- Human review on the long tail. For copy that carries high stakes (anything that lands on a homepage, anything adjacent to legal matters, anything in a regulated market) there is a human in the loop. Not because the model is bad. Because the cost of being wrong is asymmetric and you cannot evaluate your way out of asymmetric loss.
We tried, for one client, to extend their English brand voice into Bangla for the Bangladesh market, and this is the part of the project I think about most when I'm awake at 3am scrolling old Hugging Face issues. English brand voice is a problem that has been studied fairly well. There are style guides, there are corpora, and there is Strunk and White's The Elements of Style sitting on every editor's shelf in the world judging us all silently. Bangla brand voice is a different beast, and the register problem alone is enormous. Are we writing in cholito bhasha, the colloquial form, or shadhu bhasha, the literary one? Are we using English loanwords, the way most Dhaka professionals actually speak, or are we doing the Prothom Alo op-ed thing and reaching for pure Bangla equivalents that nobody says out loud over a cup of cha?
There is a DeepLearning.AI short course on multilingual RAG that gestures at this problem (worth the two hours, even if the lessons feel obvious in hindsight), but the practical answer turned out to be unglamorous. We commissioned a small original Bangla corpus, maybe two hundred pieces, written by a copywriter based in Dhaka who had also worked at The Daily Star, and we RAGged against that instead. Output quality jumped meaningfully inside two retraining cycles. The lesson I keep relearning across clients is that a small corpus of good writing beats a large corpus of bad writing on every metric that ends up mattering at QBR, and the gap is wider in Bangla than in English because the tax paid for bad translation in Bangla compounds faster. It is the kind of thing an editor at any decent Dhaka daily, or any hartal-era pamphleteer, would have told you for free over a cha in Karwan Bazar.
course2024DeepLearning.AI · Andrew Ng's shopThe multilingual RAG short course. Decent on the basics. Skip if you've already shipped multilingual; useful if you haven't.→ open sourceSo the takeaway is this. Retrieval is the easy part, and by the time you've read three blog posts and a Pinecone tutorial you can build a retrieval pipeline that demos well in a Loom. The hard parts are evals and the augmentation prompt. Evals tell you whether your system is lying, and without them you are driven by demos and you will ship a beautiful demo that quietly degrades for six months until a customer notices and your account team has a bad week and your CSM sends a passive aggressive email at 11pm. The augmentation prompt is where you encode editorial judgement. It is the editor at the magazine, the red ink on the freelancer's draft, deciding what to do with the retrieved context, which examples to honour, which rules to enforce, and when to refuse to generate. Without that prompt you have a writer with a folder of research, no deadline, and no boss.
The deeply unsexy truth of building RAG against brand voice corpora at scale is that the work that ends up mattering the most is rarely the same work that demos the best. A senior Pathao driver is not impressive because he knows where the streets are. That is table stakes any phone with Google Maps now matches in a few seconds. The reason you keep booking him over the next driver is that he knows when to peel off the main road onto the side route and when to wait the extra two minutes at the chowrasta because the warden is about to wave the other lane through. The corpus is in his memory and the editorial judgement is in his hands at the same time. Most production RAG systems I have seen, in the early months of 2026, are still operating at the Google Maps tier of that comparison. Closing the gap between that tier and the driver's instinct is where the next two years of real work happens.