I gave my agent the right memory and it ignored it anyway
A few weeks ago I was testing a support-agent setup - nothing fancy, just an LLM with a memory layer bolted on so it could remember basic facts about a user across sessions. Subscription tier, shipping address, that kind of thing.
I ran a simple scenario: the user is already on the enterprise plan. I confirmed the memory retrieval was working - the fact subscription_tier: enterprise came back correctly when I queried "what tier is the user's subscription." Then I asked the agent, in a support-chat style prompt, what plan the user was on.
The response: "Sure, upgrading to our enterprise plan would unlock that feature for you."
The user is already on enterprise. The agent had the correct fact sitting right there in its context. It just... used it wrong. Not "forgot it" - that's a different, more talked-about failure mode. This one is worse in a specific way: retrieval succeeded, the fact was injected, and the response was still confidently incorrect. Nothing failed loudly. Nothing threw an error. If I hadn't been staring at the raw context myself, I'd have had no way to know this happened except a confused (or annoyed) user telling me about it after the fact.
How Memory Frameworks Handle This
I went looking for how the popular memory frameworks handle this - Mem0, Zep, Letta, the usual suspects. They're all solving real problems: storage, retrieval, contradiction handling as facts change over time. Zep in particular does well on temporal accuracy benchmarks.
But as far as I can tell, none of them check the thing that actually broke in my test: did the LLM's response actually reflect the memory that got retrieved for it? Every framework I looked at seems to assume that once a fact is in context, the model uses it correctly. My test says that assumption doesn't always hold.
What I'm Asking the Community
So now I'm curious what other people are seeing. If you're running agents with any kind of persistent memory in production:
- Have you actually checked whether your agent uses retrieved facts correctly, or are you trusting that retrieval working means the response will be right?
- Has anyone built something that catches this - not "did retrieval return the right fact" but "did the response actually reflect it"?
- Is this a known, named failure mode I'm just not aware of, or does everyone kind of just... not check?
Genuinely asking - I've been digging into this for a bit and I'm not sure if I'm looking at something under-discussed or just late to a well-known problem.
Comments
No comments yet. Start the discussion.