Agent memory is context assembly over time

Spend a week with a coding agent and it starts to feel like it knows you. It keeps answers short because you asked once, it knows your repo conventions, and it waits at your approval gate before touching code. None of that lives in the model: when the session closes the weights do not change, and the model does not wake up tomorrow with notes about today. The remembering happens in the machinery around the model, and this post takes that machinery apart.

Memory is a harness primitive

In the harness engineering post I mapped ten primitives that wrap the model. Two of them, context management and durable state, got a paragraph each. Memory is where those two meet, and it earns its own walk: it is the set of systems that decide what an agent keeps, recalls, updates, and forgets. Said another way, memory decides what state survives long enough to shape the next action.

It sits next to RAG without being RAG. RAG asks what external knowledge should be retrieved for this answer. Memory asks what state should survive across turns, sessions, users, and workflows.

The four-type split I use below, working, episodic, semantic, and procedural, comes out of cognitive science; the CoALA paper mapped it onto language agents, and most memory products carry some version of it. As with the harness map, other people slice this differently.

One bug in a real repo

I maintain a small side project called CrikIT, a cricket league management app, and it comes with a real repo, real issues, and real session logs. I’ve simplified the details here so nothing private shows up, but the shape is what actually happened.

The job for this post is issue #9: a player profile keeps showing the player’s old club after their membership changes. The request I actually make sounds like this:

Can you pick up CrikIT issue #9?
It is the old-club-on-profile bug assigned to Buddy Bean Town.
Use the same approval-gated Codex workflow as last time.

That one request is doing a lot of work. “CrikIT issue #9” points at a specific issue thread. “Buddy Bean Town” is a human-friendly name that maps to the GitHub assignee buddybeantown. And “the same approval-gated workflow as last time” means the agent does not jump straight into editing code: I have the Codex automation set up so it only implements after someone replies @codex implement under a written proposal. Until that comment shows up, the agent stays in read-only investigation mode.

The project has standing rules too: read docs/GETTING_STARTED.md before code work, and stay inside cricketleague/ and cricketpackages/.

So the memory challenge is concrete. Can the agent combine the current request, the past issue history, the current project facts, and the right workflow, and line all of that up before it touches anything?

The session boundary

Strip everything away first. No saved state, none of the machinery that quietly loads material before the model answers. Just a model and the conversation in front of it.

If I tell it early in a chat that the Meteor app root is cricketleague/meteor/League/, and I ask about the app root a few messages later, it answers. That’s the most basic memory people experience with chat systems: the model appears to remember because the earlier message is still inside the current context.

Close that chat, start a new session, and ask it to “use the same CrikIT rule as before,” and the model has no reliable way to know what before means. It may ask a clarification, or it may guess, and a good guess is still a guess. Nothing survived the session boundary.

What the model had inside that first chat is working memory: the active context window. The latest user message, the previous replies, the instructions, and whatever files and tool results are present this turn. In cognitive science, working memory is the limited mental workspace you use for the task in front of you, and the useful part of the analogy is the limit. It runs on a budget. When people say a model has a 200K or a million token context window, they are mostly talking about a bigger working memory budget.

Working memory breaks in two ordinary ways. It ends: when the session is gone and no durable state sits behind it, the next session starts blank. And it fills up: every useful thing in the window competes with every other useful thing. Dump a whole repo, a whole issue thread, and six API manuals into context and you may technically fit the tokens and still make the model worse. It’s like emptying the whole filing cabinet onto the desk to find one sticky note. The relevant line gets buried, and the model has to pay attention across too much surface area.

So the first memory question is what should be in front of the model right now. Everything after this is about crossing the boundary that working memory cannot.

What happened: episodic memory

Episodic memory is what the agent can recall about things that happened. A specific issue investigation, a specific debugging session, the thing it learned last Friday after checking an issue thread. The key property is that it has a when.

For the CrikIT request, episodic memory is what lets the agent search past sessions:

search_sessions("CrikIT issue #9 old club buddybeantown @codex implement")

You do not want the full transcript of every CrikIT conversation dumped back. You want a focused slice:

April 25 CrikIT issue #9 run.
Issue was open and assigned to buddybeantown.
Earlier check found no explicit implementation approval.
Later check found a proposal and @codex implement.

In practical systems, episodic memory usually starts as logs: messages, tool calls, terminal output, each stamped with a time and a session ID. The storage layer can be simple. SQLite with full-text search covers a lot. A vector database helps when keyword search gets brittle, and a temporal knowledge graph, which tracks when each fact was true, helps when the order of events matters. The mechanism matters less than three questions: can the agent find the past event, can it recover the right slice, and can it tell old state apart from current state?

That last question is the limit of the type. A past session tells you what happened last time. On its own, it cannot tell you what is true right now.

What is true: semantic memory

Semantic memory is the agent’s standing knowledge. Episodic memory is pinned to a moment; semantic memory floats free. You know your team uses pnpm without replaying the pull request where that got decided.

For agents, this is where user facts, project conventions, and current state live. It’s the layer your AGENTS.md or CLAUDE.md already occupies. For CrikIT, it might contain:

CrikIT lives at ~/Projects/crikit/CrikIT.
The Meteor app root is cricketleague/meteor/League/.
Normal code work stays inside cricketleague/ and cricketpackages/.
Buddy Bean Town maps to buddybeantown.
Implementation requires proposal, then explicit approval.
Read docs/GETTING_STARTED.md before code work.

Storage is unglamorous here too: a key-value store, or a set of memory notes in markdown. Both work. The real cost is maintenance, because facts get old. An earlier run found issue #9 open but not approved; a later run found @codex implement. If both facts sit in memory with no timing and no conflict handling, the agent may retrieve the wrong one. Preferences fight the current request the same way: “keep answers short,” stored as an absolute instruction during a debugging session, becomes friction when you later ask for a full write-up. Semantic memory gives you stable knowledge and hands you three chores with it: what to save, what to update, what to retire.

How work gets done: procedural memory

Even with the facts right, the agent still needs to know how the work should be done. Procedural memory is the how-to layer: the runbook, the checklist, the skill you reuse when the same task comes back. In human terms, it is the difference between knowing a bike has two wheels and knowing how to ride one.

For CrikIT, you would not want the procedure to be “if issue #9 is open and assigned, implement it.” The real procedure carries more steps and more checks:

Fetch the issue.
Inspect the comments.
Confirm a proposal exists.
Confirm @codex implement appears after the proposal.
Read docs/GETTING_STARTED.md.
Work inside cricketleague/ and cricketpackages/.
Run focused tests.
Open a draft PR.
Comment back with the result.

“Implementation requires explicit approval” is a semantic fact. “Here is the approval-gated sequence to follow” is procedure. It can live in tool definitions, where a schema forces the agent to provide an issue number, an approval comment ID, and a branch name before it can implement. It can live in skill files and runbooks that load when the task matches. It can also live in orchestration code: workflow graphs and approval gates encode procedure even when the agent never sees the whole thing as prose.

People rarely call this layer memory. They call it tools, skills, or automation. In my book it is memory: stored knowledge about how work should happen. And it goes stale like the rest. A workflow that was fine before the project moved to a proposal-first gate becomes dangerous if the agent follows it confidently afterward.

Assembly happens every turn

In a real agent, working memory is still the same thing it was in the bare chat: the current context window, temporary, active for this turn. What changes is how it gets assembled. Before the model responds, the runtime collects material from several places: the current request, the visible conversation, a retrieved past session, current project facts, a loaded workflow, tool results, open files. All of that becomes the working context.

The durable stores are the filing cabinet and working memory is the desk. The model only ever works at the desk, so everything depends on what gets pulled out of the cabinet and set down in front of it. The model can only act on what reaches the context window.

Storing things is the easy part. The harder questions are the ones the context builder answers on every turn: what to store, when to retrieve, how much to inject, what outranks what, how to handle contradictions, and when old memory should stop helping.

Run the CrikIT request end to end and you can watch the assembly work. Working memory carries the current request. Episodic memory finds the previous issue #9 run and the change in approval state. Semantic memory supplies the repo layout, the assignee mapping, and the approval rule. Procedural memory supplies the workflow. The response you do not want is “sure, I will implement issue #9 because it is assigned.” That one walks straight past the gate. The response you want reads like this:

I found issue #9 and the prior automation history.
The earlier run was blocked because approval was missing.
The later thread includes a proposal and @codex implement.
I will read the CrikIT onboarding doc, verify the comments,
work inside the expected project roots, run focused tests,
open a draft PR, and comment back with the result.

That is what good memory looks like: recall, current facts, procedure, and a bit of judgment working together, instead of one old line played back.

Product names help here as a sanity check on the map. Hermes exposes session search, memory files, and skills, which line up almost one to one with episodic, semantic, and procedural memory. Letta, Mem0, Zep’s Graphiti, and LangChain’s long-term memory draw the same lines in their own ways. The names move around from product to product while the underlying jobs stay the same, so when you evaluate an agent product, get specific: which of these jobs is its memory actually doing?

The interesting failures are conflicts

The old event says issue #9 was open and assigned, but not approved for implementation. The current thread says @codex implement is now present. The project policy says proposal first, then explicit approval. The user’s request says pick up the old-club issue.

A weak memory system treats this as a retrieval problem: grab something related and stuff it into the prompt. A stronger one treats it as a state problem: what was true then, what is true now, which procedure governs the current action, and what should the agent verify before moving forward.

This is why “just give it a vector database” does not cover it. A vector database is great at pulling back related text. What it cannot do on its own is realize that the issue changed from non-qualifying to qualifying, that a claimed branch still needs checking, or that the user asked for the gated CrikIT workflow and not a blind code edit. The architecture has to model those differences.

Forgetting

Memory systems that remember everything eventually remember too much. Working memory fills up. Session history becomes huge. Semantic facts accumulate contradictions. Procedures drift as project rules change. An agent with perfect recall and no forgetting is the coworker who answers every question by reading you their entire diary.

Each memory type has its own version of the problem. Working memory needs selection. Episodic memory needs compression. Semantic memory needs conflict resolution, so a changed approval rule does not sit next to the old one as an equal. Procedural memory needs maintenance, so a stale “open and assigned is enough” workflow does not quietly drive code edits after the project moved to a proposal-first gate.

It’s tempting to treat forgetting as pure loss. For an agent, it is closer to hygiene: clearing out what no longer helps so the useful stuff stays easy to find. None of this means deleting everything old. Old memories can stick around. They just should not outrank what is true now.

The practical strategies are unexciting, which I take as a good sign. Temporal decay: older memories lose priority unless they are pinned, recently used, or tied to a durable policy. Contradiction handling: a new fact updates the current state and preserves the old one as history. Compression: detailed sessions become summaries, summaries become facts, and facts become procedures when they describe a repeated way of working. Manual curation: project rules, approval gates, repo onboarding, and production-facing checks are too important to leave entirely to automatic extraction. They need an owner.

This is also where the big-window argument lands. A larger context window buys more working memory, and that genuinely changes what is possible. It will not find the right past session for you, tell you which fact is current, or retire a stale runbook. Fill a million-token window with old logs and outdated procedures and all you have done is move the mess into a bigger room. The quality of context assembly matters as much as the size of the window.

Five questions

When I look at an agent now, mine or a product, I ask five questions. What is in working memory right now, meaning what did the model actually see this turn? Which past session matters, and can the agent find it? Which facts are current, and how do they get updated? Which workflow applies, and where is it encoded: tools, skills, or orchestration code? And what should be forgotten: how does old state lose priority, and who owns the procedures?

If you can answer those, you are looking at an architecture. If you cannot, you probably have scattered state, and scattered state works until the first time the agent remembers the wrong thing with confidence.

The model can only act on what reaches the context window, and the harness decides what gets there. For my own agents, most of the memory work now goes into that decision: what gets saved, what gets retrieved, and what gets retired.