Before Choosing an Agent Memory Framework

I spent some time looking through agent memory frameworks: Mem0, Zep, Letta, LangMem, and a pile of related papers and repos.

My first instinct was to compare frameworks.

Which one has the cleanest API? Which one has more stars? Which one supports graph memory? Which one has lower latency?

After a while, that started to feel backward.

Choosing a memory framework before deciding what memory means for your product is like choosing a database before knowing what data you store.

Four questions first

The useful checklist is simple:

Who is the memory for?

What should be remembered?

How long should it last?

How should it be retrieved?

If those four are clear, the technical choice becomes much easier. If those four are vague, every framework will create a different kind of mess.

Who is the memory for?

Single-user private memory is the easiest case.

Remember this user’s name, preferences, last topic, writing style, and recurring tasks. One user, one memory space, fewer conflicts.

Team memory is harder.

Now you need permissions, authorship, update rules, conflict handling, and auditability. Who can write? Who can read? What happens when two people update the same fact?

Agent self-memory is different again.

The producer and consumer are the same agent. It remembers strategies, failed approaches, successful tool sequences, and procedural habits. This is not just user profile memory. It is operational experience.

These are different products.

They should not all use the same architecture by default.

What should be remembered?

Facts are straightforward.

Preferences are softer.

Behavior patterns are experience.

Relationships often need structure.

If the memory is mostly facts and preferences, a simple store plus search may be enough. If the product needs relationship reasoning, pure vector search can fail. “A knows B” and “B works at C” does not mean A and C are semantically similar. The relation is the important part.

That is where graph memory can help.

But graph memory is not free. It adds storage, extraction, update, and query complexity.

Do not buy that complexity unless the product earns it.

How long should memory last?

More memory is not always better.

Addresses change. Preferences change. Tech stacks migrate. Project decisions get reversed.

If old memories never expire, the memory store becomes a landfill. Retrieval starts mixing stale facts with current ones, and the agent becomes confidently wrong.

Good memory needs time.

Not just created time, but validity. When was this true? Is it still true? What superseded it?

Sometimes you should not delete the old fact. You should mark it as outdated so the system can explain the change.

How should it be retrieved?

Exact lookup is simple. Key-value may be enough.

Fuzzy recall is where vector search helps.

Multi-hop reasoning is where graphs become useful.

Agent-managed memory is another path: let the agent decide what belongs in working memory, searchable recall, and long-term archival storage.

That sounds elegant, but it still needs guardrails. The agent can decide badly. The system needs logs, review, and correction paths.

My default recommendation

Start simpler than you want to.

If the product is a personal assistant with facts and preferences, use text search or vector search. Maybe even Markdown files are enough.

If the product has many entities and relationships, evaluate graph memory.

If the product needs the agent to learn procedures from feedback, treat those as versioned skills or playbooks, not vague memories.

The mistake is starting with a memory framework because it feels advanced.

Start with the memory contract.