Caching the AI: Slashing LLM Latency with Redis & MongoDB
Every LLM call is the slowest, most expensive, most rate-limited hop in your request path. Treat it like a fast database query and your portal falls over the first time a thousand users hit it at once. The fix is not a faster model — it is refusing to call the model at all when you do not have to. A disciplined cache tier between the user and the LLM is what keeps an enterprise admin portal at 99.9% uptime and 25% lower latency under real load.
The LLM is a network call you cannot trust to be fast
Provider latency is variable by design, throughput is capped by a rate limit you do not control, and every token costs money. None of those are problems you solve in the prompt — they are problems you solve in the architecture, by putting a cache between your traffic and the model and making sure the common case never reaches the provider.
A cache between the user and the model
Start with the obvious win: cache-aside on Redis. Before paying for a completion, hash the resolved prompt and check for a hit. Deterministic prompts — system summaries, classification, repeated lookups — return in single-digit milliseconds instead of seconds, and never consume rate budget.
// cache-aside: never pay for the model on a repeat
const key = hash(prompt);
const hit = await redis.get(key);
if (hit) return JSON.parse(hit); // ~3ms, $0
const answer = await llm.complete(prompt);
await redis.set(key, JSON.stringify(answer), "EX", 3600);
return answer;Exact-match is not enough — add a semantic layer
Users phrase the same question ten ways. An exact-match key misses all ten. Layer a semantic cache on top: embed the incoming prompt, search a vector index of recent prompts, and if the nearest neighbor is within a tight similarity threshold, return its cached answer. This converts a long tail of near-duplicates into hits — the single biggest lever on provider load in a high-traffic portal.
MongoDB indexing for the data the model reads
Caching the model is half the story; the other half is the retrieval the model depends on. If every completion is preceded by a slow Mongo query, you have just moved the bottleneck. Index for the actual access patterns — compound indexes on the fields your queries filter and sort by, covered indexes where you can return straight from the index without touching documents. The goal is that the data feeding the prompt is never the thing keeping the user waiting.
Insulating the rate limit at 4,000+ users
With exact and semantic caching absorbing the repeat traffic, only genuinely novel prompts reach the provider — and those you shape with a request queue and a concurrency cap tuned to your rate limit, so a spike degrades into a few hundred extra milliseconds of queueing rather than a wall of 429s. The provider never sees your peak; it sees your cache-miss rate, which is a fraction of it.
Scaling an AI feature is not about making the model faster. It is about ensuring the model is the exception in your request path, not the rule.
This is the caching architecture behind the CMZ enterprise portal — Redis cache rings, indexed reads, and rate-limit insulation holding latency and uptime steady across thousands of active endpoints.