Vector Embeddings in Production: Custom Search for RAG

Demo vector search is a single similarity query. Production vector search — the kind powering the Police RAG system over dense, multi-format legal archives — is a pipeline that has to be precise, filterable, and current. The gap between the two is where most RAG systems quietly fail: they retrieve plausible-but-wrong context and answer confidently from it.

Chunking decides retrieval quality

Before a single vector is searched, the chunking strategy has already determined the ceiling. Split mid-thought and no chunk carries a complete idea; split too coarsely and the relevant sentence drowns in noise. For legal text, chunk on semantic boundaries — clauses, sections — with a small overlap so context is never severed at the seam. Retrieval is only as good as the units you embedded.

Filter, then rank

Pure semantic similarity is not enough when correctness is legal. Combine the vector search with metadata pre-filters — jurisdiction, date, case type — so the model only ever ranks candidates that are structurally valid. This hybrid of exact filtering and semantic ranking is what keeps an autonomous system from citing an irrelevant statute that merely reads similar.

// filter on metadata, rank by meaning
const hits = await Cases.aggregate([
  { $vectorSearch: {
      index: "case_embeddings",
      path: "embedding",
      queryVector: await embed(question),
      filter: { jurisdiction, caseType },   // hard constraints first
      numCandidates: 400,
      limit: 8,
  } },
]);

Keep the index honest

An embedding index rots silently. Use one embedding model consistently — vectors from different models are not comparable — and re-embed whenever source text changes, or retrieval slowly drifts from reality. In a system that outputs verdict matrices, a stale index is not a degraded experience; it is a wrong answer with a citation attached.

In production, vector search is not "find similar." It is "find valid, then rank by meaning, and never cite what you could not ground."

For the fundamentals, see Embeddings 101 and Vector Databases for MERN Developers; the system is the Police RAG Agent.