01Everything depends on retrieval
Here is the single most important truth about RAG, and it decides whether your system is good or useless: the model can only be as right as the text you retrieved. If step one fetches the wrong chunks, no prompt, no model, no cleverness downstream can save the answer — the model literally does not have the information in front of it. So almost all of “advanced RAG” is really advanced retrieval: making sure the genuinely relevant text reaches the top of the pile.
Picture asking a junior in the library to fetch the fee notice, and watching them come back with the sports-day circular. It does not matter how carefully you then ask your question, how good your English is, or how clever the person answering is — the wrong sheet of paper is on the table, so the answer will be wrong. Everything in this chapter is about that junior: teaching them what to look for, sending them twice when one trip is not enough, and having someone check the pile before it reaches you.
user asks: “fees?”
The lazy query “fees?” embeds weakly, so the one relevant notice barely scrapes in at #4 — and if you only send the top 3 to the model (to save tokens), it never sees the answer at all.
02Fix one: better chunking and metadata
The cheapest wins are before you retrieve at all. Revisit chunking from Chapter 9: chunks split mid-idea retrieve badly, so split at natural boundaries and overlap them. And attach metadata to every chunk — which document, what date, which category — because then you can filter before you search: “only search notices from this semester” or “only the fees category”. Narrowing the haystack before looking for the needle is often the biggest single improvement, and it is pure Chapter 4 thinking applied to vectors.
03Fix two: query expansion
Users type terrible queries — “fees?”, three words, no context. A tiny embedding of “fees?” is a weak, vague position (Chapter 8) that matches poorly. Query expansion fixes this by improving the query before searching: use a cheap, fast LLM call (Chapter 6) to rewrite “fees?” into “When is the tuition fee payment deadline and how much is it?” — a richer query whose embedding lands much closer to the right chunks. A variant generates several rewordings and searches with all of them, catching more of the relevant text. You are spending one small model call to make retrieval far sharper — a trade that almost always pays.
04Fix three: hybrid search
Chapter 9 gave you two kinds of search and made you pick semantic. The truth is you often want both. Semantic search nails meaning but can miss an exact token — a product code “CS-402”, a specific name, an error number — because to an embedding those are just odd strings. Keyword search nails the exact token but misses meaning. Hybrid search runs both and merges the results, so “the CS-402 lab timing” finds the chunk that contains the exact code and is about lab timings. Most serious RAG systems are hybrid; pure semantic is the teaching version.
05Fix four: re-ranking
The most powerful upgrade, and worth understanding well. Vector search is fast but approximate (Chapter 9) — it is good at getting the relevant chunks into the top twenty, but not perfect at ordering them, so the best chunk might come back fourth. Feeding the model twenty chunks is wasteful (Chapter 6 — tokens cost) and can bury the answer.
Re-ranking adds a second, more careful pass: retrieve wide, then sharpen. Fetch the top 20 candidates cheaply with vector search, then run them through a stronger, slower model — a re-ranker — whose only job is to score how well each chunk actually answers this specific query, and keep the best 3 or 4. Those go to the LLM. It is the same “cast a wide net, then inspect carefully” instinct a good researcher uses, and it is often the difference between a RAG demo and a RAG product.
06Diagnose before you fix
The senior skill here is not knowing the four fixes — it is knowing which one a given failure needs, and that comes from looking. When an answer is wrong, do not guess: print what was retrieved. If the right chunk is not in the retrieved set at all, your problem is retrieval — reach for query expansion, hybrid, or better chunking. If the right chunk was retrieved but ranked low and got crowded out, that is a re-ranking problem. If the right chunk was there, ranked first, and the model still answered wrong, now it is a prompt problem (Chapter 7). A professional reads the pipeline’s middle before changing its ends. Amateurs randomly swap models and hope.
07Do this today
Take your Chapter 10 RAG and add exactly one upgrade — query expansion is the easiest big win. Before searching, make a cheap LLM call that rewrites the user’s short question into a fuller one, then search with that. Ask a deliberately lazy question (“fees?”) before and after, and print the retrieved chunks each time. Watching the right notice climb into the results because you improved the query, not the model, is the whole lesson of this chapter in one experiment.