LLM Memory Architectures: A Taxonomy
LLM Memory Architectures: A Taxonomy
v1.0 | 2026-07-31 | Status: Draft
Overview
Large Language Models don’t have memory in the human sense, but we can build memory systems around them. The current landscape of approaches falls into roughly five categories, ordered by distance from the model’s native operation:
1. In-Context Memory
Definition: Memory that fits entirely within the model’s context window.
Mechanisms:
- Direct prompt injection
- Chain-of-thought with history
- Few-shot examples as memory
Limitations:
- Context window size (typically 4k-128k tokens)
- No persistence between sessions
- Token cost scales with memory size
Best for: Short conversations, single-session tasks
2. Retrieval-Augmented Memory
Definition: Memory stored externally, retrieved and injected into context as needed.
Mechanisms:
- Vector databases (FAISS, Pinecone, Weaviate)
- Semantic search over memory corpus
- Hybrid keyword + vector retrieval
Patterns:
- RAG (Retrieval-Augmented Generation)
- Memory as embeddings + metadata
- Chunked document retrieval
Limitations:
- Retrieval quality depends on embedding model
- No temporal reasoning about memory
- Hallucination still possible
Best for: Knowledge bases, long-form Q&A, document-grounded responses
3. Graph-Based Memory
Definition: Memory structured as nodes and relationships.
Mechanisms:
- Knowledge graphs (Neo4j, NetworkX)
- Entity relationship extraction
- Path-based reasoning
Patterns:
- Personal knowledge graphs
- Conversation as graph traversal
- Multi-hop reasoning
Limitations:
- Schema design complexity
- Graph query language overhead
- Scaling challenges
Best for: Structured knowledge, entity relationships, multi-turn reasoning
4. Process-Based Memory
Definition: Memory as code or executable processes.
Mechanisms:
- Programming language execution
- API calls to external systems
- Tool use with memory side effects
Patterns:
- Function-calling LLMs
- Agent architectures with tool access
- Memory as database writes
Limitations:
- Security concerns (code execution)
- Debugging complexity
- Determinism challenges
Best for: Automated workflows, system integration, action-taking agents
5. Meta-Memory (Memory of Memory)
Definition: Memory about the memory system itself.
Mechanisms:
- Reflection on memory usage
- Confidence scoring for retrieved memory
- Memory provenance tracking
- Self-modifying memory structures
Patterns:
- Memory confidence tags
- Provenance chains
- Adaptive retrieval strategies
- Self-correcting memory
Limitations:
- Meta-overhead can exceed value
- Complexity explosion
- Evaluation difficulties
Best for: High-stakes applications, long-lived agents, trust-critical systems
Synthesis
No single approach is optimal for all use cases. The most effective systems combine multiple layers:
User Query
↓
[Meta: Select retrieval strategy]
↓
[Retrieval: Find relevant context]
↓
[Graph: Extract entity relationships]
↓
[In-Context: Format for LLM]
↓
LLM Response
↓
[Process: Take actions, update memory]
This multi-layer approach is what I’m building into Aurora.
Open Questions
- How to balance retrieval quality vs. latency?
- Can we build memory systems that feel like human memory (associative, imperfect, evolving)?
- What’s the right abstraction level for memory APIs?
- How do we evaluate memory system quality?
Related
- [[Aurora architecture]] - How these concepts apply to my agent project
- [[Physics of memory]] - Analogies from statistical mechanics
- See also: Memory in the age of LLMs (external link)
Version history: v1.0 (2026-07-31) - Initial draft