Memory & Stateful AI: Beyond the Context Window

Memory is what separates an agent that knows you from a chatbot that forgets you between messages. The context window is RAM, not disk — small, expensive, and wiped every request — so a stateful agent can't just keep appending the transcript. It tiers its memory: a working buffer for the current turn, an episodic store of conversation summaries, and a semantic store of durable facts — and it recalls only the slice each turn actually needs.

The context window is RAM, not disk

Treating the window as storage fails twice. It overflows — a long relationship will not fit, and naive truncation amputates the oldest, often most important, context. And it's costly — every token of stuffed history is paid for on every single turn. The fix is the fix every computer uses: a small fast tier you compute over (the prompt) backed by larger slow tiers you fetch from on demand. You don't hold the whole history in context; you retrieve the relevant part of it.

Tier by lifetime: turn, episode, fact

Different memories have different half-lives. The current exchange lives in the working buffer — verbatim, in the prompt, gone next turn. Older exchanges get compressed into the episodic store as vector-indexed summaries you can semantically retrieve when something relevant comes up again. And stable truths — the user's name, preferences, account tier — get promoted to a semantic store of durable facts, looked up by key, never summarized away. Recall pulls from the lower tiers into the prompt; it never loads everything.

// each turn: pull only relevant memory, answer, then write back
const facts = await semantic.get(userId);          // durable, by key
const episodes = await episodic.search(embed(turn), 4); // relevant summaries
const prompt = assemble({ facts, episodes, turn }); // small, targeted

const res = await model.generate(prompt);

await episodic.upsert(summarize(turn, res));        // compress this exchange
await semantic.merge(userId, extractFacts(res));    // promote stable truths

Compression is a first-class step, not an afterthought

Memory that only grows is memory that eventually breaks — retrieval gets noisier and costs climb. So the write path matters as much as the read path: after each response, summarize the exchange before storing it, and extract any durable facts into the semantic tier. Summarization is lossy on purpose — it's the forgetting that keeps the system usable, the same way human memory keeps the gist and drops the wording. An agent that remembers everything is as broken as one that remembers nothing.

Statefulness isn't a bigger context window — it's a memory hierarchy. Keep the turn in RAM, summaries and facts on disk, and recall only what this moment needs. The window is where you think, not where you store.

Memory is the layer that makes stateless edge agents feel continuous, and its episodic tier is just vector retrieval pointed at the conversation instead of the docs — bounded by the same context budget every prompt lives under. Continue on the roadmap.