AI Agent Memory Architecture: What to Remember, and What to Forget

An agent without memory asks the customer the same questions every session. An agent that remembers everything replays thousands of tokens of old conversations on every call, gets slower and more expensive each week, and starts acting on things that stopped being true months ago. Memory needs a design.

Two kinds of memory

Working memory

Everything in the context window during one run: the request, the conversation so far, tool results. It is the agent's short-term memory and it has a hard size limit. Manage it actively: summarise older turns, trim tool output to the fields that matter, and drop what the current step does not need.

Long-term memory

What survives between runs. Store it as small, structured facts, not transcripts. "Brand tone: plain, no exclamation marks." "Prefers calls after 3pm." "Offer changed to annual billing on October 4." Each fact has a type, a value, a source and a date.

How the agent writes memory

Give the agent explicit memory tools: remember a fact, update a fact, forget a fact. Do not try to extract memory from every transcript after the fact. When the agent learns something worth keeping, it calls the tool, and code validates and stores it. In the page builder I run, memory is kept per brand and written through tools like these, so each fact has a clear origin.

How the agent reads memory

At the start of a session, load a short summary of the relevant facts into the context, a few hundred tokens at most. Give the agent a tool to look up more if it needs specific detail. This keeps every call small while the agent still benefits from history.

Forgetting is a feature

• Overwrite, do not append. When a fact changes, update it rather than storing both versions.
• Let people correct it. A forget tool, and a screen where a user or admin can see and delete facts.
• Expire time-bound facts. A promotion that ended should not shape next month's copy.
• Keep memory scoped. One customer's facts must never load into another customer's session.

What not to do

• Do not store full conversations as memory and reload them.
• Do not put memory in the system prompt in a way that changes every call, which breaks prompt caching. Load it after the stable prefix.
• Do not let the model write free-form notes with no structure. They become impossible to update or audit.

Memory is one of the five parts of every agent, covered in AI agent architecture, and a major input to context engineering.

A memory schema that works

Here is a simple shape for long-term memory facts:

At session start, code loads the facts for the scope, drops expired ones, and renders them as a short summary grouped by type. The agent has tools to remember, update and forget facts by key.

Types of AI agent memory

• Working memory: the current context window.
• Episodic memory: what happened in past sessions, stored as short summaries.
• Semantic memory: durable facts and preferences, as above.
• Procedural memory: how to do things, which usually lives in instructions and the knowledge base rather than per-user memory.

Memory in agent tools and frameworks

Workflow tools such as n8n offer simple session memory for chat agents, and frameworks offer memory stores. They are useful starting points, but check what they store. Many keep raw message history, which grows forever. For anything long-lived, add structured facts with overwrite and forget.

How I use it

The page builder on my systems page keeps memory per brand as structured facts, written through tools and loaded as a summary. The self-selling bot keeps conversation memory and syncs key details to the CRM, so the CRM remains the system of record.

Memory and privacy

Memory is personal data. Scope every fact to one user or account, never load facts across scopes, record where each fact came from, and give people a way to see and delete what is stored. Do not store secrets such as passwords or full card numbers in agent memory at all. These rules are easy to build in at the start and painful to add later.

Testing memory

• Tell the agent a fact, start a new session, and confirm it uses the fact.
• Change the fact and confirm the old version is gone.
• Ask it to forget and confirm the fact no longer loads.
• Run two accounts side by side and confirm nothing leaks between them.

Frequently asked questions

How does memory work in AI agents?

Agents have working memory, which is the current context window, and long-term memory, which is an external store of facts that persists across sessions. Long-term facts are loaded as a short summary at the start of a session and updated through tools.

What should an AI agent remember?

Small, durable, structured facts: preferences, decisions, key account details and constraints, each with a source and date. Not full transcripts, and not details that only mattered in one conversation.

How do you stop agent memory from growing forever?

Overwrite facts when they change instead of appending, expire time-bound facts, summarise working memory as it grows, and give users or admins a way to view and delete stored facts.

What are the types of AI agent memory?

Working memory (the current context), episodic memory (summaries of past sessions), semantic memory (durable facts and preferences) and procedural memory (how to do tasks, usually held in instructions and knowledge bases).

Where should AI agent memory be stored?

In a database outside the model, as structured records scoped to a user, account or brand. For business data that already has a home, such as contact details, the CRM should stay the system of record.

Is AI agent memory a privacy risk?

It can be. Scope memory to one user or account, never load it across scopes, record the source of each fact, avoid storing secrets, and give users a way to view and delete stored facts.