Semantic Caching That Actually Hits: Matching Meaning at the Edge

Running the streamerOS support agent taught me that support traffic is gloriously repetitive. "How do I connect my OBS?", "how do I link OBS", "OBS setup help" — three phrasings of one question, and a naive RAG pipeline runs the full retrieve-and-generate gauntlet for every single one. Each is a model call I paid for and a second of latency the user waited through, to produce an answer already produced a hundred times. Semantic caching fixes this by recognising the question, not the string.

Why a normal cache misses

A key-value cache keyed on the raw query string is useless here, because no two users phrase a question identically. The cache hit rate hovers near zero. What you actually want to match on is meaning — and you already have the machinery to measure meaning, because your RAG pipeline embeds the query anyway. A semantic cache reuses that embedding to ask a different question: have I already answered something that means this?

The mechanism

Embed the incoming query — you were going to anyway. Before doing any retrieval or generation, search a small vector index of previously answered queries. If the nearest neighbour sits above a similarity threshold, return its stored answer and stop. If not, run the full pipeline, then write the new query embedding and its answer back to the cache. The cache learns the shape of your traffic over time, and the hot questions go nearly free.

// check meaning, not string equality
const qVec = await embed(query);
const [near] = await cache.query({ vector: qVec, topK: 1 });

if (near && near.score >= 0.95) {
  return near.metadata.answer;        // HIT — no retrieval, no LLM
}

const answer = await runRagPipeline(query, qVec);
await cache.upsert({ vector: qVec, metadata: { answer } });  // learn it
return answer;

The threshold is a product decision

The similarity floor is the one dial that matters, and it is not an engineering constant — it is a risk choice. Set it high (0.97+) and you only ever reuse answers to near-identical questions; safe, lower hit rate. Set it lower and you catch more paraphrases but risk serving a confidently adjacent answer to a subtly different question. For a grounded support agent I keep it conservative, because a wrong cache hit undoes the entire zero-hallucination guarantee.

Invalidation is the catch

A cached answer is a snapshot of the documentation at the moment it was generated. When the knowledge base changes, stale entries become a liability — they will happily serve last month's instructions. The clean fix is to tag every cache entry with the corpus version it was derived from and drop the whole cache on re-index. The cache is cheap to rebuild; serving a confidently outdated answer is not.

Semantic caching is the rare optimisation that improves cost and latency at once — you are not making the model faster, you are skipping it entirely for questions you have already answered.

This sits in front of the edge-native pipeline and respects the grounding contract — a cache hit is only valid if the original answer was grounded. It is the kind of trade-off I weigh on every system I ship: the cheapest, safest win is usually the work you can avoid doing at all, and knowing where that line sits is most of the job.