Conversation Memory: There Is No Magic Here

The truth about memory

Lesson 1 said it: the model is stateless. Every call starts from zero. So "memory" is nothing more than a list of messages you keep and resend.

Here is a complete memory implementation:

history = []

def chat(user_input):
    history.append({"role": "user", "content": user_input})
    resp = llm.invoke(history)
    history.append({"role": "assistant", "content": resp.content})
    return resp.content

chat("My name is Maya.")
print(chat("What's my name?"))     # Maya

The second call works because history now holds four messages, and the model can read the first two. Comment out the first chat(...) and the same code answers "I don't know your name."

That's it. That's memory. Everything else in this lesson is engineering detail: where to store the list, how to keep users apart, and what to do when it gets too long.

The LangChain version

So why not just use the ten lines above? Because that history is a module-level global shared by every user of your server, and because you'd have to hand-manage it in every chain you write. The LangChain version fixes both:

from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a patient programming teacher. Keep answers brief."),
    MessagesPlaceholder(variable_name="history"),   # past messages get spliced in here
    ("user", "{input}"),
])

chain = prompt | llm | StrOutputParser()

store = {}

def get_history(session_id: str):
    if session_id not in store:
        store[session_id] = InMemoryChatMessageHistory()
    return store[session_id]

chat_with_memory = RunnableWithMessageHistory(
    chain,
    get_history,
    input_messages_key="input",
    history_messages_key="history",
)

cfg = {"configurable": {"session_id": "user-42"}}
chat_with_memory.invoke({"input": "My name is Maya."}, config=cfg)
print(chat_with_memory.invoke({"input": "What's my name?"}, config=cfg))

The second invoke prints something like "Your name is Maya." Change session_id to "user-99" on that line and it will tell you it doesn't know — which is exactly the isolation you want.

Three things to understand:

• MessagesPlaceholder is the slot where history goes, and its position matters. It has to sit after the system message and before the current user message, so the model reads the conversation in chronological order with the rules established first.
• session_id is the key that keeps users apart. In a real product it must be a user ID or a conversation ID. Hardcoding it means every user on your site shares one conversation, and user A can read user B's messages. Nothing crashes — it just quietly leaks. That's a privacy incident, not a bug.
• InMemoryChatMessageHistory lives in process memory, which means it vanishes on restart and isn't shared across server instances. For production, swap in a database or Redis implementation. The community packages ship RedisChatMessageHistory and SQLChatMessageHistory; check current docs for the import path, since these have moved between packages more than once.

Compare this with the ten-line version and you can see the trade precisely. You wrote more code. In exchange you got per-session isolation, a pluggable storage backend, and a chain that still supports .stream(). That's the deal LangChain always offers: structure in exchange for ceremony. Take it when you need the structure.

Context will overflow

The longer the conversation, the more history rides along on every request. Which means: steadily more expensive, steadily slower, and eventually an outright error when you cross the model's context limit.

Three responses, in increasing order of complexity.

1. Sliding window — keep only the most recent N turns. Simplest thing that works, and it's enough for the large majority of applications.

from langchain_core.messages import trim_messages, HumanMessage, AIMessage

msgs = [HumanMessage("one"), AIMessage("two"), HumanMessage("three"), AIMessage("four")]
trimmed = trim_messages(msgs, max_tokens=2, strategy="last", token_counter=len)
print([m.content for m in trimmed])    # ['three', 'four']

Using token_counter=len counts messages instead of tokens, which makes the behavior easy to see and easy to test without a model. In real use you pass the model itself as the counter and set max_tokens to something like 2000.

2. Summarization — compress older turns into a paragraph. Saves tokens, loses detail, and each summary costs an extra model call.

3. Window plus summary. Keep the last N turns verbatim, summarize everything before. Best results, most moving parts.

Start with option 1. Move to option 3 only when users actually complain that the assistant forgot something. Building the complex version first is a reliable way to spend a week on a problem you didn't have.

Don't use memory as a database

Here's a mistake that looks reasonable and isn't: stuffing structured facts — the user's plan tier, their order status, their account balance — into the conversation history so the model will "remember" them.

Don't. Chat history gets trimmed. It gets summarized. The model sometimes skims it. None of those are acceptable behaviors for a balance figure.

Structured facts belong in a database. Query them when you need them and inject the current values into the system prompt:

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a support agent.\nCustomer plan: {plan}\nOpen orders: {orders}"),
    MessagesPlaceholder(variable_name="history"),
    ("user", "{input}"),
])

Now the facts are fresh on every single turn, they can't be trimmed away, and if the order ships mid-conversation the model sees the new status.

Memory is responsible for conversational coherence. The database is responsible for correctness. Keep that line sharp and you'll avoid a whole family of strange production bugs where the assistant confidently states something that was true twenty minutes ago.

Ten-second check

What happens if you ship with session_id hardcoded to "default"?

Every user shares one conversation history — user A can see user B's chat. It won't raise an error, which is exactly what makes it dangerous.