Building Conversational AI With Memory
On this page
Most chatbots have amnesia. Close the tab, come back tomorrow, and you are a stranger again. That gap between a stateless language model and a genuinely helpful assistant is almost entirely a memory problem. A large language model (LLM) generates text from whatever you put in its context window — nothing more. If you want it to remember your name, your preferences, or what you argued about three sessions ago, you have to build that memory yourself.
This post walks through how conversational memory actually works, the architecture patterns that hold up in production, and the practical decisions you will face when you build one.
Why Language Models Need External Memory
An LLM is a pure function: tokens in, tokens out. It has no persistent state between calls. The illusion of memory in a chat interface comes from one simple trick — the entire conversation history is re-sent with every request.
That works until it doesn't. Context windows, even large ones, are finite and increasingly expensive as they fill. Re-sending a 40-turn conversation on every message wastes tokens, adds latency, and eventually overflows. Worse, models suffer from "lost in the middle" degradation: information buried in a long context gets less attention than information at the edges. Stuffing everything into the prompt is not the same as remembering it well.
External memory solves this by storing information outside the context window and retrieving only what is relevant for the current turn. The model stays lean; the knowledge scales independently.
The Three Layers of Conversational Memory
It helps to think about memory in three distinct layers, each with different lifetimes and retrieval strategies.
Short-term (working) memory is the current conversation. This is the raw turn-by-turn exchange, usually kept verbatim in the context window up to some budget. It is cheap to manage and high-fidelity — the model sees exactly what was said.
Long-term (episodic) memory is what survives across sessions. Facts the user told you last week, decisions you made together, the running history of the relationship. This lives in a database and gets retrieved on demand.
Semantic memory is distilled knowledge rather than raw transcript. Instead of storing "I'm allergic to peanuts" as a chat message, you extract and store the structured fact user.allergies = [peanuts]. Semantic memory is compact, queryable, and far more reliable to act on than a fuzzy search over old messages.
A robust system uses all three. Short-term keeps the conversation coherent, episodic gives continuity, and semantic gives the assistant a durable model of who it is talking to.
Storage Patterns That Work
Once you decide to persist memory, you need somewhere to put it. Three approaches dominate, and most mature systems combine them.
Summarization Buffers
The simplest durable pattern. When the conversation grows past a token threshold, summarize the oldest turns into a compact paragraph and drop the originals. You keep a rolling summary plus the last few verbatim turns. This is cheap and keeps context bounded, but summarization is lossy — details get smoothed away, and you cannot recover them later.
Vector Stores and Retrieval
For episodic recall, embed each message or memory chunk into a vector and store it in a vector database. At query time, embed the user's new message and retrieve the most semantically similar past memories. This is the "RAG for memory" pattern, and it scales to enormous histories.
# Store a memory
embedding = embed(message.text)
vector_db.upsert(id=message.id, vector=embedding, metadata={
"user_id": user_id,
"timestamp": message.timestamp,
"text": message.text,
})
# Retrieve relevant memories for the current turn
query_vec = embed(current_message)
memories = vector_db.query(query_vec, top_k=5, filter={"user_id": user_id})
context = "\n".join(m.metadata["text"] for m in memories)
The catch: pure semantic similarity retrieves things that are related, not necessarily important or recent. A good retrieval layer blends similarity with recency and an importance score, so that "the user's wedding is next month" outranks a semantically similar but trivial aside.
Structured Fact Stores
For semantic memory, skip the vectors and use a plain database or key-value store with an extraction step. After each turn — or in a background job — run an LLM pass that pulls out durable facts and writes them to structured records. Retrieval becomes a straightforward query rather than a fuzzy search, which makes the assistant's behavior far more predictable.
A Reference Architecture
Putting it together, a request flows like this:
- Ingest the new user message into short-term memory.
- Retrieve relevant long-term memories — a hybrid query over the vector store (semantic) and the fact store (structured), scoped to this user.
- Assemble the prompt: system instructions, retrieved memories, rolling summary, and the recent verbatim turns.
- Generate the response with the LLM.
- Write back asynchronously: append to short-term memory, extract new facts, and update embeddings — ideally off the critical path so the user does not wait.
The write-back step is where most teams cut corners and later regret it. Extracting facts synchronously adds latency; skipping extraction entirely means your semantic memory never grows. A background queue is the right answer.
Practical Advice From the Trenches
Scope every memory to a user (and often a session). Cross-contaminating one user's memories into another's context is both a correctness bug and a privacy breach. Make user_id a mandatory filter on every retrieval, not an afterthought.
Give memories a decay and a review path. People change jobs, move cities, change their minds. Timestamp everything, prefer recent facts on conflict, and let users see and delete what the system remembers about them. A visible memory is a trustworthy one.
Budget your context explicitly. Decide in advance how many tokens go to retrieved memory versus recent conversation versus system prompt. Retrieve top-k, not top-everything. More context is not more intelligence past a point — it is more cost and more noise.
Deduplicate before you write. Naive systems store "user likes tea" fifteen times. Check for near-duplicate facts on write and merge or update instead of appending.
Measure retrieval quality directly. The failure mode of memory systems is silent: the model answers plausibly using the wrong or missing context, and you never notice. Log what was retrieved for each turn and periodically evaluate whether the right memories surfaced. Retrieval precision is your real health metric, not response fluency.
Handle the cold start gracefully. A brand-new user has no memory. Make sure your prompt assembly degrades cleanly to "no prior context" rather than injecting empty placeholders or apologizing for not remembering.
Prefer extraction over raw transcript recall for anything you will act on. Searching old messages is fine for "remind me what we discussed." But if the assistant needs to do something based on a fact — book the vegetarian option, ship to the new address — pull it from a structured store you can trust.
Common Pitfalls
The biggest one is treating memory as pure vector search. Semantic similarity is powerful but blind to importance, recency, and truth. A conversation full of casual chatter will drown a single critical fact under a pile of similar-looking noise. Layer in scoring.
The second is unbounded growth. Without summarization, decay, or deduplication, memory stores bloat, retrieval slows, and relevance drops. Prune deliberately.
The third is ignoring privacy and consent until it becomes a compliance fire. Build the delete path and the "what do you remember about me" view early. They are far cheaper to design in than to retrofit.
FAQ
Do I still need a memory system if my model has a huge context window? Yes. A large window lets you carry more, but it does not persist anything between sessions, does not scope data per user, and gets expensive and less accurate as it fills. Large contexts and external memory are complements, not substitutes.
Vector database or relational database? Both. Use a vector store for fuzzy episodic recall ("what did we talk about?") and a relational or key-value store for structured facts you act on. The hybrid is more work but far more reliable than either alone.
When should I extract facts — inline or in the background? Background, almost always. Synchronous extraction adds a whole extra LLM call to your response latency. Queue the transcript and process it after the response is sent, unless a fact is needed for the very next turn.
How do I stop the assistant from hallucinating memories? Retrieve with confidence thresholds and pass memories as clearly labeled context ("Known facts about this user: ..."). Instruct the model to say it does not know rather than invent. And never let it treat an absence of retrieval results as license to guess.
How much history should I keep verbatim? Typically the last 5–15 turns, tuned to your token budget, plus a rolling summary of everything older. The exact number depends on your model's window and your latency and cost targets — measure it rather than guessing.
Is this the same as RAG? Architecturally similar — both retrieve relevant text and inject it into the prompt. The difference is the source. RAG retrieves from a static knowledge base; conversational memory retrieves from a growing, per-user, time-sensitive store that you write to on every interaction.
Closing Thought
Memory is what turns a clever text generator into something that feels like a continuous relationship. The model does the reasoning, but the memory layer decides what the model gets to reason about. Get the layering, scoping, and retrieval scoring right, and the assistant stops being a stranger every morning — and starts being genuinely useful.