RAG — making the model answer from your documents

01The problem, stated one last time

Chapter 5: the model is frozen and knows nothing after its training cutoff, and nothing about your private data. Chapter 7: prompting can reduce hallucination but cannot give the model knowledge it never had. So how does a support bot answer questions about a product released last month, or a college assistant answer from this year’s notices? Not by retraining the model — that costs a fortune and you would do it every time a notice changed. There is a far cleverer way.

The insight is almost embarrassingly simple. The model is a brilliant reader and writer; it just lacks your facts. So do not try to put your facts inside the model. Instead, at the moment of the question, hand the model the relevant text and ask it to answer using only that — the way you would give a sharp intern the right file and say “answer from this”. That is RAG: Retrieval-Augmented Generation. Retrieve the right text, augment the prompt with it, generate the answer.

02Walk the pipeline

And you already built every piece. RAG is not a new technology — it is Chapters 8 and 9 wired into Chapter 6. Step through it:

Read that again and notice something wonderful: there is no magic box. Vector search (Chapter 9) finds the relevant chunks; the prompt (Chapter 7) stuffs them in with an instruction; the LLM (Chapter 6) writes the answer. RAG is a pattern, not a product — an architecture (Chapter 1!) that connects tools you already understand. That is exactly why understanding the foundations mattered so much.

03The whole system on one page

The left half is the indexing from Chapter 9, done ahead of time. The right half is the live path: question in, retrieve, augment, generate, answer out. The same vector database sits in the middle, filled once and queried forever. If you can draw this diagram from memory and explain each arrow, you can pass most “explain RAG” interviews on the spot.

04The augmented prompt — where grounding happens

The heart of RAG is one carefully built prompt. After retrieval, you assemble a message that puts the fetched text in front of the question, with a strict instruction — this is Chapter 7 doing the most important work of its life:

SYSTEM:
Answer the student's question using ONLY the notices below.
If the answer is not in them, say exactly: "I don't have that information."
Never use outside knowledge. Cite the notice you used.

CONTEXT (retrieved by vector search):
[1] "Last date for tuition payment is Friday 14th."
[2] "Scholarship application window closes on the 30th."

QUESTION:
When is the fee due?

That instruction — “using ONLY the notices below” — is called the grounding contract, and it is the line that turns a confident liar into a trustworthy assistant. It aims the model at real, retrieved words instead of its foggy memory, so its answer is anchored to your actual documents. Everything good about RAG lives in that one instruction being present and obeyed.

05Citations and refusal — the trust features

Two behaviours separate a toy RAG from one people trust with real decisions, and you can now see why each works. Citations: because you stored the source text with every vector (Chapter 9, the habit I made you build), you know exactly which chunk fed the answer — so you can show “from Notice dated 3rd” beside it. A cited answer is one a human can verify, and that changes everything about whether they trust it.

Refusal: the “if it is not in the notices, say I don’t have that” clause. It sounds like a weakness; it is the opposite. A system that honestly says “I don’t know” for the 5% it cannot answer is infinitely more deployable than one that invents fluent nonsense for all 100%. Refusal is what makes RAG safe to put in front of real users — and it is only possible because retrieval gives the model a defined set of text to be “in” or “not in”.

06When basic RAG disappoints — and the fixes

Build the simple version and you will hit its limits, so here are the real-world upgrades (Day 10 of the live sessions) — each a targeted fix for a specific failure, not random cleverness.

The retrieval missed the right chunk. Everything depends on step one fetching the relevant text; if it does not, the model cannot answer no matter how good it is. Fix the retrieval: better chunking (Chapter 9), or query expansion — quietly rewrite or enrich the user’s short query before searching, so “fees?” becomes something a search can actually match.

Keywords and meaning both matter. Pure semantic search can miss an exact code or name; pure keyword search misses meaning. Hybrid search runs both and merges the results — Chapter 9’s two halves, working together.

The right chunk came back ranked fourth. Vector search is fast but approximate. Re-ranking adds a second, more careful pass: fetch the top 20 candidates cheaply, then use a stronger model to re-order them and keep the best 3 to send to the LLM. Retrieve wide, then sharpen. Together these turn a demo that works sometimes into a system that works reliably — and being able to name which fix solves which failure is exactly the senior skill.

07Testing the untestable

One hard truth from Day 12: how do you test a system whose answers are written fresh each time and can be phrased a thousand ways? You cannot check for an exact string like a normal test (Chapter 3). So AI teams build an evaluation set — a collection of real questions with their known-correct answers and known sources — and measure the system against it: did retrieval fetch the right chunk? Did the answer match the truth? Did it correctly refuse when the answer was absent? You will not perfect this as a student, but knowing that serious AI work is measured, not vibed is what separates a professional from a demo-maker.

08This is a real, shipped architecture

Everything in this chapter is exactly how production AI features are built today — a support bot over a company’s help docs, a legal assistant over case files, a “chat with your PDF” tool. The concierge that answers questions on this very site is a RAG system: it retrieves from a fixed corpus and refuses politely when a question falls outside it. You are not learning a toy. You are learning the single most employable AI pattern of this era, from the same foundations the real ones are built on.

09Do this today

Turn the semantic search you built in Chapter 9 into a system that answers. On a query: retrieve the top three notices (you already have this), drop them into the grounding prompt from section 04, and send it to your LLM. Now ask your notice board a real question in plain language and get back a sentence that answers it from your actual notices, with the source shown. Then test its honesty: ask something no notice covers, and watch it refuse instead of inventing. When it does — when your own system says “I don’t have that information” instead of lying — you have built the thing companies are hiring people to build.

You can now ground a model in real knowledge. The final step is to let it not just answer, but act — take steps, use tools, do things in the world. That is agents, and it is next.