Agent Memory Management
Design short-term and long-term memory that helps agents without accumulating unreliable state.
Memory is stored state
Agent memory is not a mystical ability to remember. It is information stored somewhere and selected for a later request. Conversation context is short-term state supplied in the prompt. Long-term memory may be a database of preferences, prior tasks, summaries, or facts. Each category needs an owner, retention rule, access policy, and deletion path. Once memory is treated like data, familiar questions become visible: who wrote it, when was it true, where did it come from, and who may read it?
Short-term context
Conversation history helps an agent resolve references and maintain continuity, but sending everything forever increases token cost and distraction. Keep a bounded recent window, summarize older turns when useful, and preserve important identifiers separately from prose. Summaries can omit or distort details, so retain links to original records for verification. Remove secrets and unnecessary personal data before storing context. The current prompt should contain only information needed for the next decision, not an accidental archive of the user’s life.
def retrieve_memory(records, user_id, purpose, limit=3):
eligible = [
record for record in records
if record["user_id"] == user_id
and purpose in record["allowed_purposes"]
and record["confidence"] >= 0.8
]
eligible.sort(key=lambda record: record["updated_at"], reverse=True)
return eligible[:limit]
memories = retrieve_memory(memory_store, "user-42", "study-plan")
Long-term memory types
Separate durable preferences, factual profile data, episodic task history, and derived summaries. “Prefers concise emails” is different from “invoice 482 was paid on Tuesday,” and both differ from a model-generated guess about the user’s priorities. Different types require different confidence, expiration, and correction rules. Store provenance, timestamp, scope, and source. If a memory cannot be explained or corrected, it should not silently influence an important action. Every category should have a clear retention owner.
Write selectively
Do not save every message or let the model directly promote arbitrary text into permanent memory. Extract candidate memories, check whether the user intended them as durable, validate their format, and apply a policy before writing. A user saying “for this task, use a formal tone” may be temporary rather than a lasting preference. Let users inspect, edit, and delete memories. Selective writes reduce noise, privacy risk, and the chance that a one-off statement becomes a persistent instruction.
Retrieve with limits
Memory retrieval should be a filtered search, not a full dump. Apply tenant, user, purpose, and permission filters before relevance ranking. Limit the number and size of returned memories, prefer recent or high-confidence items, and show their source when the decision matters. Treat recalled text as untrusted data because an attacker may have inserted instructions into a note or imported document. The application should decide which memories are eligible; the model should not grant itself access.
Correctness and forgetting
Facts change, people change their minds, and memories can be wrong from the moment they are written. Add timestamps, expiration dates, confidence, and conflict handling. When two memories disagree, ask for clarification or use an authoritative system of record instead of choosing whichever text sounds more recent. Implement deletion by identifier and verify that caches, summaries, indexes, and backups follow the product’s retention policy. Forgetting is a feature that must work across every representation of the data.
Example: a study coach
A study coach might remember that a learner prefers examples before definitions, the topics already practiced, and a target exam date. It should not permanently remember a guess that the learner is weak at a subject based on one incorrect answer. Progress belongs in a structured learning record, while style preference can be user-editable memory. Before recommending a plan, the coach can retrieve a few relevant records, explain the basis of the recommendation, and ask the learner to correct outdated information.