A few words about RAG

A quick rundown of what RAG (Retrieval-Augmented Generation) actually is.

Since LLMs have no long-term memory, and the context window is nowhere near big enough to hold every document you need, someone came up with the following: you take the user's query and, through a series of tricks (for example: "Here is a query, guess what the answer to it might look like" - yes, really, that one is called HyDE) and other sleight of hand, turn it into a database query (and usually against a vector database or, God help us, a graph one).

You get back a list of documents, most of which bear only a passing relation to the query, sort them, filter them (with the same LLM), and stuff them into the context. The LLM then gazes wisely at the garbage delivered to it and produces nonsense. Garbage in, garbage out.

The trouble is not retrieval quality. The trouble is that this is a fixed pipeline: one pass, one set of results, no feedback. Whatever got retrieved is what you live with.

On a small, clean corpus anything works, which is why the client demo comes out great. The trick is to make your escape over the horizon in time, right after the rollout.

The idea of just using grep is not exactly original, but it is often misunderstood. You do not need grep as such, you need an agentic loop that has both exact string search and fuzzy pattern matching. If you are sitting on a terabyte of documents, grepping them is a bad idea: take Elastic, ts_vector, or even vector embeddings if you have images.

The tool is not the point at all. The point is that instead of a fixed pipeline, the agent decides for itself what to search for next and how, based on what turned up the last time around.

That is roughly how we search Google: we change our approach depending on what we find, rather than typing a keyword and clicking the first link. And note that there is a very expensive index underneath. The loop belongs on top of good search, not in place of it.

The usual objection: an agentic loop is slow and expensive. Agreed, and let me add a third: it is also unreliable, a weak model simply cannot pull it off. This is an architecture for strong models, not a way to save money on them.

But the alternative is fast, cheap, and crap.