Vector Foundations: How Semantic Search Actually Works

Keyword search asks "does this string appear?" Semantic search asks "does this mean the same thing?" The bridge between those two questions is the embedding: a model that turns a span of text into a vector, positioned so that things which mean similar things land near each other. Get this layer right and "reset my password" finds "recover account access" without sharing a single word.

Meaning becomes geometry

An embedding model maps text into a high-dimensional space where distance encodes similarity. You never look at the 1,536 numbers directly — you compare them. Cosine similarity between two vectors approximates how related their meanings are, which means search becomes a geometry problem: find the nearest neighbours to the query vector.

ANN indexes trade exactness for speed

Comparing the query against every vector (a brute-force k-NN scan) is exact but doesn't scale — at a million documents it's a linear wall. Approximate nearest-neighbour indexes like HNSW and IVF give up a sliver of recall to return results in sub-millisecond time, by navigating a graph or probing a subset of partitions instead of scanning everything. For production retrieval, "approximately the right ten, instantly" beats "exactly the right ten, eventually."

// embed the query, then ANN-search the store
const [qVec] = await embed([query]);

const hits = await store.search({
  vector: qVec,
  topK: 8,                 // recall vs. token-budget tradeoff
  metric: "cosine",
  filter: { lang: "en" },  // metadata pre-filter narrows the space
});

Chunking decides quality more than the model does

Teams obsess over which embedding model to use and under-invest in how they split documents. But retrieval quality is dominated by chunk design: too large and a chunk dilutes its own meaning across many topics; too small and it loses the context that made it answerable. The unit you embed isthe unit you retrieve — design it for the question you expect, not for the convenience of your parser.

Semantic search isn't a better keyword index. It's a different question — "what means this?" — answered by turning meaning into coordinates and asking what's nearby.

This vector layer is the retrieval half of RAG, and the place where semantic caching later earns its keep. Continue on the roadmap.