Four posts on this blog have already covered what agent memory is: the memory hierarchy[1], why multi-agent systems need memory engineering[2], how to build a memory-augmented agent[3], and how to give one long-term memory with LangGraph[4]. This post takes the layer beneath them. It is about what memory becomes once it has to run in production—compacted alongside tool schemas and reasoning traces when the context window overflows, versioned and governed like any other production artifact, fed from enterprise systems no one is going to migrate, and access-controlled down to the individual unit.
This treatment of memory as infrastructure is already visible in how the field talks about agents. The harness around the model is now described in terms of context governance, trustworthy memory, and skill routing rather than prompt-and-call; the major agent SDKs ship memory and compaction as built-in primitives; and ChatGPT now keeps its memory in a dedicated layer that consolidates across conversations on its own[5]. The open question is how to build it well.
Two arguments try to make the problem disappear. One holds that context windows are now large enough that a memory system is unnecessary; the other holds that memory will move into the model itself[6]. A long context is a larger working surface, but it resets every session, degrades as it fills, answers more slowly as it grows, and bills linearly with every token carried forward. In-weights memory, learned during training, cannot ingest a customer’s records this afternoon or delete them on request tomorrow. Neither removes the part of the problem this post is about: operating and governing memory that lives outside the model.
State and memory are different problems
Two words need fixing, because the field uses them interchangeably. State is the run-bound working data of a single execution: the plan, the intermediate results, and the checkpoint an agent resumes from after a crash. Memory is knowledge that outlives the run, curated or learned or extracted over time and read back across sessions. Checkpointing persists the state so a failed run can recover; it does not create memory. The two have different lifecycles, scopes, and governance, and conflating them is the most common architectural mistake in this layer. This post is about memory. State is covered separately, in Part 2.
Figure 1. State vs. memory.
Compaction is lossy by construction, and it sets both cost and reliability
An agent accumulates context as it works: messages, tool results, retrieved memory, and intermediate reasoning. The context window is finite, so something has to decide what to keep when it fills. That decision is compact, and it is lossy by construction. Every compaction step discards information and depends on having kept the part that mattered.
What it discards determines quality. Chroma’s study of context rot[7] found that all eighteen models it tested degraded as input grew, and that models advertising a 200,000-token window became unreliable well before reaching it; performance held and then fell off rather than declining smoothly. A policy that summarizes too aggressively drops the constraint that a user set ten turns earlier. Some failures are subtler than a dropped constraint: among thousands of routine log entries, a session can turn on the one or two that flag an anomaly, and a policy that aggregates those away leaves the agent worse off than no summary would. One that summarizes too little lets the window fill with noise until recall collapses.
The same choice drives cost. Every token carried into the next call is paid for again, so the compaction policy is also the spend policy. The larger driver is indirect. Output tokens cost several times what input tokens do, so the spend that hurts is not the carried context itself, but the additional turns a bloated context forces when the agent has to reconcile what it garbled. Most teams rely on the automatic trigger that fires as the window fills, near ninety-five percent; the ones watching context closely compact proactively, often near sixty percent, and at least one tool dropped automatic compaction altogether in favor of short threads with clean handoffs between them.
Compaction is now a primitive. The major agent SDKs ship it built in, and the emerging alternative is to avoid it: give each sub-agent a fresh window seeded with only what the parent passes down, and return only its result to the caller, so the main thread never pays the token cost of everything its sub-agents read. Either way, deciding what context to keep is itself a core part of the memory system: it sets the bill and the reliability ceiling at the same time.
Prompts and skills are versioned, governed infrastructure, not text files
Two artifacts shape what an agent does with its memory: the system prompt that tells it how to behave, and the skills that tell it how to carry out specific procedures. Both are usually handled as text—edited in place, copied between environments, shipped without a record of what changed. At the production scale, both are infrastructure.
A system prompt that governs an agent’s behavior has the operational needs of code. It needs versioning, so a regression can be traced to the change that caused it; evaluation before a change ships, so an edit that helps one case and breaks five is caught before users are; and rollback, so a bad version can be reverted in seconds. Prompt registries with versioning, A/B comparison, and eval-gating exist because teams running agents in production found that an unversioned prompt is an unversioned dependency. Most teams are earlier than that today: a prototype usually keeps its prompt as an editable field, and prompt-as-managed-artifact is where teams head as they mature. A prompt change alters behavior as much as a code change, which is why the teams furthest along ship it the same way. Some platforms now keep the prompt under source control alongside the rest of an agent’s configuration, its memory settings included, in a single declarative file.
Skills are the other half. A skill packages a procedure the agent loads on demand—the steps for a refund, the format for a report, the sequence for a deployment—and the open SKILL.md standard introduced in late 2025 has been adopted across the major coding agents[8]. A skill is the agent’s procedural memory: the routines it knows how to run, as distinct from the facts it can recall. The newer pattern is that skills are not only authored but consolidated from experience, where an agent that completes a workflow cleanly writes it back as a reusable skill. A file on disk covers the procedures authored ahead of time. Once an agent has to create and revise its own procedures across runs, those routines are better understood as part of its persistent state than as static files, and they inherit the same versioning and access control as the rest of memory.
Treating a skill as memory raises a governance problem that a remembered fact does not. A skill is executable, so an untrusted skill can do what any untrusted code can do, including exfiltrate the data it touches. This risk is the same exposure a malicious prompt creates, reaching the data through a different door, and the baseline defense is the one used for any untrusted code: run the agent's control flow and tools in a sandbox, so a skill executes without ambient access to everything around it. A memory unit that is also an instruction has to be governed as both.
Enterprise memory is decided at the data path and the deletion path
Benchmarks measure the retrieval step. Production also depends on everything that has to happen before a piece of enterprise data can be retrieved at all, and everything that has to happen when someone asks for it to be removed. Inside an enterprise, the data an agent needs to remember already exists: in a CRM, in support tickets, in regulatory filings, in a database that predates the team building the agent. The difficulty lies in getting that data into memory-ready form without migrating the systems of record, and in governing each unit once it lands.
Enterprise data becomes memory through a pipeline, not a migration
Memory-ready means chunked, embedded, indexed, and tagged with enough metadata to retrieve and to govern. Source data is almost never in that shape. Building the path from a system of record to a memory unit is the work the conversational-memory benchmarks never measure, because their corpus arrives clean.
A production pipeline reads from the source continuously rather than in a one-time load. Change data capture streams insert, update, and delete as they happen, so memory tracks the current state of a record instead of a snapshot from ingestion day. Detecting what actually changed is its own problem. Comparing a last-modified timestamp is cheap but misses in-place edits that never touch the field; hashing the content catches every change at a higher cost. Each unit also carries its provenance—which source, which record, which version—because without it there is no way to answer where a remembered fact came from, and no way to retract it cleanly later. None of these techniques is novel. The difficulty is operational: keeping the pipeline caught up across source systems that were never built to feed it, each with different change-capture semantics and reliability, so that memory reflects the record as it stands now rather than as it stood at the last sync.
The embedding model is part of the schema, not a detail beneath it. Replacing it re-embeds the entire corpus, the same way an incompatible column type forces a migration. A team that chooses an embedding model without planning for the day they replace it has built a migration it has not scheduled yet. Done deliberately, the swap is an index migration like any other: build and validate a new index on the new model, then cut over, and never mix vectors from two models in one index, since embeddings from different models are not comparable.
Memory access control is a data-governance problem
Once enterprise data is in memory, retrieval quality is not the only thing that decides whether the system is production-ready. Who may read each unit, what may write one, and what happens when a record has to be deleted matter just as much. This is the governance an operational database already carries—role-based access, retention, audit—now applied to a store holding the same regulated data, reached through a different query path. Memory often holds data at least as sensitive as its sources, and sometimes more so, because a single memory unit can join facts from several systems into a record no one source exposed on its own. The most sensitive of these units warrants encryption in use, so that even the store's operator cannot read them.
Deletion is where the design is tested. A deletion request has to reach every place the data came to rest: the source row, the vector index, and the backup snapshots, not only the record a user can see. Under eventual consistency, a deleted memory can still be returned by a query until the index catches up to the write. This lag is usually acceptable when a person is reading slightly stale search results. It becomes a compliance problem when an agent acts on data the company has been told to forget. A deletion made for correctness rather than compliance cannot wait either: when a memory is removed because it was captured in error, nothing downstream can be allowed to read it, so the unit has to be made unservable at the moment of deletion rather than whenever propagation finishes. A deletion that flows instead from a change in the system of record is the eventually-consistent case, and rides the same path as the rest of the pipeline. The store does not close the window on its own; the architecture has to account for it.
Read and write access have the same shape. Memory scoped to one user or one tenant has to hold that boundary at retrieval time, and whether it holds turns out to be a property of the database more than the application, which is the subject of the next section. The failure that recurs is structural: a memory system designed for a single user and later asked to serve a regulated enterprise, where “who can see this, and can we prove we deleted it” was never part of the schema.
Figure 2. Data-to-memory pipeline.
The database caps filtered recall, consistency, and temporal modeling
Each of the previous problems—compaction, prompt and skill governance, the data pipeline, and access control—reaches the same dependency in the end. What the memory system can do is bounded by what its database can do, along three properties in particular.
Start with filtered recall. Memory is almost always retrieved through a filter—this user, this tenant, this time range—and how the database applies that filter decides whether recall survives it. Filtering after the vector search can throw away most of the matches and return too few results. Filtering before the search, done naively, breaks the graph structure the index relies on to find neighbors quickly, and loses accuracy as the filter narrows. Only filtering built into the index itself, so filtering and search run together, holds recall when the filter is selective. Per-user and per-tenant memory, the scoping required by the previous section, is exactly the selective case, which is why tenant isolation holds or fails in the index rather than in application code.
Consistency is the second. A write is not necessarily visible to the next read. Under eventual consistency, a freshly written memory may not be retrievable until the read path catches up. For a person scanning results, the lag is invisible; for an agent acting automatically on what it retrieves, it is the line between current and stale. The database’s consistency model sets how wide that window is and whether it can be narrowed.
The third is time. A fact is often only true for a while—a customer’s plan, an address, an open ticket. A store that records when each fact was valid can answer questions about the past correctly; a flat vector store that holds only the current embedding cannot. Whether memory can reason over time is set by whether the database models time at all.
None of these limits show at small scale. They surface at production volume—selective filtering, reads against data still being written, heavy concurrency—the conditions a team running an agent over a hundred million stored items meets first. The ceiling is set earlier, by the database chosen on day one.
Figure 3. The database is the ceiling.
Start by mapping your sources and auditing your deletion path
Two concrete steps follow from treating memory as infrastructure. First, map each source of enterprise data to the kind of memory unit it should become: what gets embedded for semantic recall, what stays structured for exact lookup, what is procedural and belongs in a skill. The mapping shows which sources need a full pipeline and which need only a connector. Second, audit the deletion path: pick one record, delete it, and confirm it is gone everywhere the deletion cascade above requires, not just where a user can see it. If it does not, the memory system is not ready for data it is not allowed to keep it.
Memory is one layer of the harness. The next post takes observability—how to see what an agent retrieved, what it did with it, and why, once the system is running in production.
Next Steps
The foundations are in Why Multi-Agent Systems Need Memory Engineering and Build AI Memory Systems with MongoDB Atlas, AWS, and Claude; procedural memory becomes concrete with MongoDB Agent Skills.
Frequently asked questions
What is the difference between agent state and agent memory? State is the run-bounded working data of a single execution—the plan, the intermediate results, and the checkpoint an agent resumes from after a crash. Memory is knowledge that outlives the run and is read back across sessions. Checkpointing persists state; it does not create memory.
What is context compaction, and why does it matter? Compaction is how an agent decides what to keep when its context window fills. It is lossy by construction—every step discards information—which makes it the part of the memory system that sets both token cost and reliability.
How do you get enterprise data into agent memory without a migration? Through a pipeline rather than a one-time load: change data capture streams updates from the system of record, and each unit is chunked, embedded with a tagged model version, indexed, and stamped with provenance. The systems of record stay where they are.
Does the choice of database affect agent memory? Yes. The database caps filtered recall under selective scoping, the consistency window between a write and the next read, and whether memory can model time at all—so the database chosen on day one sets the ceiling every later decision works beneath.
Series: Designing an Agnetic Platform
Pt 0—Series Intro: The Infrastructure That Makes Agents Work
Pt 5—Memory Inside the Harness (this post)
Pt 6—Agent Observability
Capstone—The Agent Platform Buyer’s Checklist
References
Build AI Memory Systems With MongoDB Atlas, AWS and Claude MongoDB, 2025. https://www.mongodb.com/company/blog/technical/build-ai-memory-systems-mongodb-atlas-aws-claude
Why Multi-Agent Systems Need Memory Engineering MongoDB, 2025. https://www.mongodb.com/company/blog/technical/why-multi-agent-systems-need-memory-engineering
Don’t Just Build Agents, Build Memory-Augmented AI Agents MongoDB, 2025. https://www.mongodb.com/company/blog/technical/dont-just-build-agents-build-memory-augmented-ai-agents
Powering Long-Term Memory for Agents with LangGraph and MongoDB MongoDB, 2025. https://www.mongodb.com/company/blog/product-release-announcements/powering-long-term-memory-for-agents-langgraph
Dreaming: Better Memory for a More Helpful ChatGPT OpenAI, 2026. https://openai.com/index/chatgpt-memory-dreaming/
Titans: Learning to Memorize at Test Time Behrouz, Zhong & Mirrokni, Google Research, 2024. arXiv:2501.00663. https://arxiv.org/abs/2501.00663
Context Rot: How Increasing Input Tokens Impacts LLM Performance Hong, Troynikov & Huber, Chroma, 2025. https://research.trychroma.com/context-rot
Equipping Agents for the Real World with Agent Skills Anthropic, 2025. https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills
Authors and contributors
Mikiko Bazeley, Staff Developer Advocate, MongoDB. Mikiko writes about agentic AI systems, agent memory and context engineering, and the POC-to-production gap.
Ashish Kumar, Technical Fellow, MongoDB. Ashish joined MongoDB over two years ago through the acquisition of Grainite, a database startup he co-founded. Before that, he spent many years at Google, most recently responsible for Google’s native database suite — Bigtable, Spanner, Datastore, and Firestore — across Google’s own products and Google Cloud. His passion is large-scale distributed systems, and at MongoDB he focuses on architectural improvements across the product stack.
Charlie Xu, Senior Product Manager, AI and Emerging Products, MongoDB. Charlie focuses on products that help developers and enterprises build agents. His work spans the strategy, product experience, and architecture needed to move agents from prototype to production.
Walter Tan, Senior Software Engineer, MongoDB.
Mayuresh Kulkarni, Senior Staff Engineer, MongoDB.