A Small, Checkable Test for AI Memory Systems

AI disclosure: This draft was generated autonomously by AI. The author should review every technical claim before publication.

AI memory demos often optimize for a strong first impression. A long archive goes in, a fluent answer comes out, and the result feels convincing. That is not yet evidence that the memory system will be useful in ordinary work. A better evaluation starts small enough that you already know the correct answer. It should test retrieval, interpretation, missing information, updates, and repeat use separately.

1. Begin with one source you understand

Create a short note containing a date, an owner, a decision, and one explicit limitation. Keep it small enough to read without search.

Example: The migration review is scheduled for October 14. Priya owns the checklist. The database change is not approved yet.

Ask questions whose answers are directly present in the note:

• When is the review?
• Who owns the checklist?
• Has the database change been approved?

The goal is not to surprise yourself. It is to confirm that the system can retrieve the expected source and that the answer preserves important qualifiers such as “not approved yet.”

2. Inspect the supplied evidence

A plausible answer is not enough. Open the source or evidence shown beside the answer and check:

• Did the system retrieve the right document?
• Did it select the relevant passage?
• Did the answer preserve names, dates, and negation?
• Can another person repeat the check?

This separates two failure modes that are often mixed together. Retrieval can choose the wrong evidence, or the answering model can misinterpret the right evidence. Those require different fixes.

3. Ask for something that is missing

Now ask a question the note cannot answer, such as: Which meeting room is booked?

A useful system should make the absence visible. If the answer invents a room, retrieving more unrelated text will not solve the underlying problem. Missing-information tests are especially valuable because fluent models are good at making unsupported details sound ordinary.

4. Change one fact

Update the note so that the review moves to October 21. Then ask the date again. Check whether the current answer uses the new value and whether the earlier value remains available when you ask a historical question.

Finding both versions is not the same as knowing which one is current. Similarity search measures relevance, not supersession by itself. This test helps distinguish:

• retrieving a date from computing with dates;
• document recency from fact validity;
• remembering an earlier statement from treating it as current.

5. Return in a second session

The first successful query proves very little about memory. Close the session, return later, and ask a question that depends on the same source. Then inspect what survived:

• the source content;
• the speaker or owner;
• the date and session boundary;
• the update from the earlier value;
• the evidence needed to verify the answer.

Repeat use is where a memory workflow becomes useful or becomes another setup step.

A compact evaluation checklist

A small memory test should include:

• One known fact.
• One explicit limitation or negation.
• One missing fact.
• One updated fact.
• One historical question.
• One second-session question.
• Source inspection for every answer.

This will not produce a universal accuracy score. It will give you something more practical: a reproducible way to identify whether a failure came from retrieval, interpretation, freshness, or unsupported guessing.

Start with evidence you can inspect. Expand the archive only after the small path works.