AI Agents Don’t Need More Context. They Need Memory.

We keep making AI models better at reasoning. We give them larger context windows. We connect them to tools. We let them search files, browse repositories, call APIs, execute code, and operate increasingly complex workflows. And yet one problem keeps showing up: the agent forgets. Not necessarily because the model is bad. Because most agent systems still treat memory as an afterthought. I’ve been thinking about this problem for a while, and it eventually led me to start building BaseMyAI: a local-first memory infrastructure layer for AI agents. This is the first post where I want to document what I’m building, why I think this problem matters, and what I’m learning along the way. Context is not memory A common approach to agent memory looks roughly like this: - Store conversations or documents. - Generate embeddings. - Put them in a vector database. - Retrieve the closest chunks for the next prompt. This is useful. But I don't think it is memory. It is retrieval. A real memory system has to answer harder questions. What does the agent currently believe? What information is outdated? Which fact replaced another fact? Which memories belong to this agent? Which memories are temporary? Which ones must survive for months? What happened before a certain decision? What information is actually relevant to the current task? And just as importantly: what should be forgotten? Once agents start operating for days, weeks, or months, these questions become much more important than simply finding the nearest embedding. Bigger context windows don't solve this Long context windows are incredible. But throwing everything into the prompt doesn't scale particularly well. Imagine an engineering agent that has worked on the same codebase for six months. During that time it has seen: - thousands of commits; - architecture decisions; - abandoned implementations; - bug investigations; - conversations; - documentation; - benchmarks; - temporary hypotheses; - user preferences; - tool outputs. Technically, you could keep feeding more information back into the model. But eventually you're paying for a huge amount of irrelevant context. And worse: old information can conflict with new information. The problem becomes less about: “How much context can the model read?” and more about: “What is the smallest amount of correct context the model needs right now?” That is a memory problem. Memory needs time One concept I find particularly important is temporal memory. Consider these two facts: Database: PostgreSQL Database: native embedded engine A basic retrieval system might return both. But they're not necessarily contradictory. Maybe PostgreSQL was used three months ago and the project later migrated to a native engine. The missing dimension is time. The system should understand something closer to: 2026-04 Database = PostgreSQL 2026-07 Database = native embedded engine Now an agent can reason about the evolution of the project instead of treating every stored fact as equally current. That distinction becomes extremely important in long-running software projects. Memory needs boundaries Another problem appears when multiple agents are involved. Imagine: coding-agent research-agent support-agent marketing-agent They may share some knowledge. But they should not automatically share everything. An agent's memory needs an identity and a boundary. This raises interesting architecture questions around: - isolation; - permissions; - shared memory; - provenance; - ownership; - synchronization. For BaseMyAI, agent isolation is one of the fundamental primitives rather than something added later. Local-first changes the architecture There is another requirement I care about: memory should be able to live close to the user. Agent memory can contain some of the most sensitive information on a machine: source code, conversations, documents, product strategy, credentials metadata, personal preferences, and months of accumulated context. Sending all of that to another hosted database should not be the only architecture available. So BaseMyAI is being designed around a local-first and encrypted model. That decision makes the engineering considerably more interesting. I'm currently building a native storage engine in Rust with things like persistent indexes, bounded memory management, WAL durability, snapshots, compaction, and concurrency controls. The goal isn't to build infrastructure for the sake of infrastructure. The goal is to make long-term agent memory predictable enough that developers can actually trust it. A vector database is still useful None of this means vector search is bad. Vector search is extremely useful. BaseMyAI itself uses vector retrieval as one part of memory. The distinction I'm making is architectural: Vector search ↓ is a component of ↓ Agent memory rather than: Vector database = Agent memory Memory also needs structure, lifecycle, chronology, identity, durability, and context selection. That's the layer I'm interested in. The direction I'm exploring My current mental model looks something like this: ┌─────────────────┐ │ AI Agent │ └────────┬────────┘ │ ▼ ┌─────────────────┐ │ Context Compiler│ └────────┬────────┘ │ ┌───────────────┼───────────────┐ ▼ ▼ ▼ Semantic Temporal Structured Recall Memory Relations │ │ │ └───────────────┼───────────────┘ ▼ ┌─────────────────┐ │ Durable Memory │ └─────────────────┘ The important component here might actually be the context compiler. The storage engine can know millions of things. The model shouldn't receive millions of things. The context compiler's job is to transform long-term memory into a small, relevant, current representation for a particular request. I'm increasingly convinced that this layer will be critical for serious autonomous agents. Building this in public BaseMyAI is still being built. A lot of the work right now is deep infrastructure work rather than polished product work. Rust. Storage engines. Memory accounting. Concurrency. Indexes. Durability. Retrieval. Temporal semantics. And probably many design decisions I'll discover were wrong six months from now. That's exactly why I want to write about it here. Instead of only publishing BaseMyAI once everything looks finished, I want to document the engineering decisions, experiments, failures, benchmarks, and architectural questions as they happen. Some topics I want to explore next include agent memory models, temporal retrieval, designing a storage engine in Rust, context compilation, memory isolation between agents, and why BaseMyAI is deliberately not designed as another vector database. If you're working on agents, retrieval systems, Rust infrastructure, knowledge graphs, or long-term AI memory, I'd genuinely like to compare approaches. This field still feels very early. And I think we're only beginning to understand what memory for software agents should actually look like. Top comments (0)