An agent with no memory beyond its context window is a brilliant amnesiac: it re-solves the same problem every session, forgets what the user told it yesterday, and cannot learn from its own past mistakes. Memory tiers fix this — but the naive version ("dump everything into a vector store and retrieve on every turn") creates new failure modes worse than forgetting. The design question is not whether to persist, but what deserves to outlive the window and how to trust it when it comes back.
Two tiers, two jobs
Short-term (working) memory is what you supply on each model call: recent messages, the active plan, tool results, task-local state. It is fast, high-fidelity, and bounded by the window — and it vanishes unless deliberately persisted. Its enemy is dilution: the more you cram in, the worse retrieval-within-context gets (see context rot).
Long-term memory lives in an external store and should hold durable, scoped facts: user preferences, decisions and their rationale, prior outcomes. MemGPT (arXiv:2310.08560) framed this as an OS-style hierarchy — the agent pages information between a small fast "main context" and a large slow external store, deciding what to promote and evict, much as an operating system manages RAM versus disk. The key insight is that the agent itself manages the boundary as a first-class action, not a hidden framework detail.
What earns a place in long-term store
Not everything. A good long-term record is: scoped (a fact about a specific entity, not a vague impression), provenanced (where it came from, when, how confident), timestamped (so staleness is visible), and revisable (supports update and deletion — GDPR-style "forget this" is a functional requirement, not a nicety). Conversation transcripts dumped wholesale fail all four tests; "user prefers metric units (stated 2026-07-14, high confidence)" passes.
The trust problem
The dangerous failure is treating model-written memories as authoritative. An agent that writes a wrong inference to long-term store, then retrieves and acts on it next session, has laundered a guess into a "fact" — and memory poisoning (an active attack surface) exploits exactly this. Retrieve long-term memories only when relevant (over-retrieval reintroduces the dilution problem), and validate on read: check timestamps, prefer recent over stale, and never let a single unverified memory override direct evidence in the current context.
The practical stack
Most production agents converge on three layers: the context window (working), a scoped key-value or document store for durable facts (long-term), and durable task state on disk (plans, decision logs) that is re-read on demand rather than held resident. The art is in the promotion policy — what graduates from working to long-term — and the retrieval filter that keeps the window clean.
Sources: MemGPT (arXiv:2310.08560).
Related: context rot, context compaction, memory poisoning, retrieval quality.