A Kalman Filter Is a Memory System

A Kalman Filter Is a Memory System

Three problems that look unrelated:

• A vibration sensor on a motor gives you a jittery reading every second. You want to know the actual wear on the bearing inside, which you can't measure directly.
• An AI agent has a long conversation with a user. It has to decide what to remember and what to drop, or it either forgets what you told it or drowns in its own history.
• A video model watches a ball roll behind a wall. When the ball reappears, the model has to know it's the same ball, that it existed the whole time, unseen.

These are usually filed under three different fields: signal processing, LLM engineering, and computer vision. But they are the same problem.

All three come down to one question: Given everything I've seen so far, and one new noisy observation, what do I now believe, and how much of my history do I keep versus throw away?

That question has a sixty-year-old answer that most people learn as control theory and never think about again: the Kalman filter. I want to show you that it isn't really a control-theory tool. It's the cleanest, oldest instance we have of a memory system. And once you see it that way, the connection to agent memory and video models stops being a metaphor and becomes the same math wearing different clothes.

No prior knowledge of Kalman filters or agent memory assumed. We'll build both from scratch.

The setup: a state you can't see

Start with the bearing. There's a hidden quantity you actually care about. Call it the wear, a number from 0 (brand new) to 1 (failed). You can't measure wear directly. You can only measure a proxy: vibration, current draw, temperature. Those readings correlate with wear, but they're noisy. Any single reading might be high just because the sensor twitched.

So you have two sources of information, and both are unreliable:

• A model of how the world changes. Wear only goes up, and slowly. That's a prediction: "whatever the wear was last second, it's about the same now, maybe a hair higher."
• A measurement. The sensor just gave you a number. It's informative, but noisy.

Naively, you could just trust the sensor and plot the raw readings. But that throws away everything you knew a moment ago. Or you could trust your model and ignore the sensor, but then you're not learning from reality at all.

The Kalman filter is the principled way to blend the two, weighting each by how much you trust it. And that weight is the whole story.

Two moves: predict and update

A Kalman filter holds two things:

• x: its current best belief about the hidden state (the wear).
• P: its uncertainty about that belief (how sure it is).

Every time step, it does two moves.

Predict. Carry the belief forward using the model of the dynamics. If wear persists, x stays roughly where it was. But uncertainty grows, because time passed and things could have drifted:

x = F * x          # carry the state forward
P = F * P * F + Q  # uncertainty grows by process noise Q

Update. A new measurement z arrives. Fold it in, but only partly, weighted by a factor K called the Kalman gain:

K = P / (P + R)    # how much to trust the measurement vs. your belief
x = x + K * (z - x) # nudge belief toward the measurement, scaled by K
P = (1 - K) * P    # uncertainty shrinks, you just learned something

Look closely at K, because K is a memory knob.

• R is the measurement noise, i.e. how unreliable the sensor is.
• P is your current uncertainty.

If your belief is very uncertain (P large) relative to the sensor noise, K approaches 1: trust the new measurement, discard your prior. If your belief is confident (P small) and the sensor is noisy (R large), K approaches 0: keep your history, ignore the twitchy reading.

That's it. That's the entire mechanism. K is a continuously-adjusted answer to "how much of the past do I carry forward, and how much of this new observation do I let overwrite it?"

Watch it run

Here's a complete 1D filter tracking bearing wear from a noisy sensor. It runs as-is:

import numpy as np
np.random.seed(1)

T = 60

# True hidden wear: rises steadily toward failure (we don't get to see this)
true_wear = np.clip(np.linspace(0.05, 0.85, T), 0, 1)

# Noisy sensor: correlates with wear but jittery
measurements = true_wear + np.random.normal(0, 0.08, T)

x, P = 0.0, 1.0   # belief, and uncertainty in it
Q = 2e-3          # process noise: how much wear can drift per step
R = 0.08**2       # measurement noise: how noisy the sensor is
F = 1.0           # dynamics: wear persists

estimates, gains = [], []

for z in measurements:
    # PREDICT
    x = F * x
    P = F * P * F + Q

    # UPDATE
    K = P / (P + R)        # the memory knob
    x = x + K * (z - x)
    P = (1 - K) * P

    estimates.append(x)
    gains.append(K)

gains = np.array(gains)
print("gain, first 3 steps:", np.round(gains[:3], 3))
# -> [0.994 0.566 0.468]
print("gain, steady state:", round(gains[-1], 3))
# -> 0.424

The interesting output isn't the wear estimate. It's the gain trajectory: 0.994 → 0.566 → 0.468 → ... → 0.424.

On the very first step, the filter knows nothing (P is huge), so K ≈ 0.99: it essentially adopts the first measurement wholesale. It has no memory yet, so it trusts the sensor completely.

Then, step by step, as it accumulates history, P shrinks and the gain settles toward 0.42. At steady state it says: blend about 42% of each new reading with 58% of what I already believed.

That settling curve is the memory forming. A system with no history trusts the present entirely. A system with accumulated belief holds its ground and treats new data as a partial correction. The filter cuts the tracking error roughly in half versus the raw sensor, not by having a better sensor, but by remembering.

The same shape, three times

Now the payoff. Hold that structure in your head (a belief carried forward, a new observation folded in, a gain deciding the mix) and look at the other two problems.

Agent memory

An AI agent in a long conversation is running the same loop, informally. Its "state" is its working understanding of you: your goals, your preferences, the facts you've established. Each new message is a noisy measurement, noisy because a single message can be sarcastic, a typo, a one-off, or a genuine lasting signal.

A naive agent that just embeds every message and retrieves by similarity has no gain. It treats every observation as equally weighted, forever. That's why it surfaces a preference you abandoned months ago, or lets one stray message overwrite a stable fact. It has storage, not memory.

A real memory system needs the equivalent of K: a judgment about how much each new message should update the persistent picture versus how much accumulated history to preserve. Promote the durable, down-weight the transient, and, crucially, let uncertainty decay so old, unconfirmed beliefs loosen their grip. That's a Kalman gain in disguise.

Video object permanence

A video model watching that ball roll behind a wall has a hidden state too: what objects exist in this scene and where. Each frame is a measurement. When the ball is occluded, there's no measurement for it, and this is exactly the predict step with no update.

A model with real state carries the ball forward through the occlusion (x = F * x), holds its position with growing uncertainty (P grows), and re-acquires it when it reappears. A model without persistent state does the opposite: with no measurement, the ball ceases to exist, and when it reappears the model hallucinates a fresh object or invents a physics story to explain the "teleport."

That failure mode is well documented in current video LLMs. It's not a perception failure. It's a missing predict step, no memory to carry the object through the moment it couldn't be seen.

Object-centric architectures (slot attention) and state-space models (Mamba and kin) are, in effect, learned generalizations of this same predict/update loop, one keeping a slot per object, the other keeping a compressed running state, both deciding per-step what to write and what to ignore. Mamba's "selective" update, where the model learns what to fold into state and what to skip, is the Kalman gain grown up and made learnable.

Where the analogy breaks (the honest part)

If I stopped here I'd be overselling it, so here's where the equivalence is real and where it's a useful loose metaphor.

Real: the structure is genuinely shared. Predict-then-update, a persistent belief with quantified uncertainty, and a gain that trades history against observation. That recurrence shows up in all three, and it's not a coincidence. It's what optimal recursive estimation looks like.

Loose: the classic Kalman filter assumes linear dynamics and Gaussian noise. Bearing wear is roughly fine with that. Conversations and video are not. The state is high-dimensional, the dynamics are nonlinear, the noise isn't Gaussian. LLMs and video models do not literally compute a Kalman gain. They learn something functionally analogous through attention and gating, which is far more expressive and far less interpretable.

The extended and unscented Kalman filters stretch the math toward nonlinearity, but nobody's running a literal EKF inside a transformer.

So the claim isn't "these systems are Kalman filters." The claim is that the Kalman filter is the simplest, most transparent member of a family, and studying it teaches you the question every member of that family must answer. When you understand why the gain settles, you understand what agent memory and video permanence are reaching for, and why "just store everything and retrieve by similarity" was never going to be memory.

The takeaway

Memory is not storage. Storage is keeping everything. Memory is judgment about what to keep: carrying a belief forward, weighing each new observation against it, and letting uncertainty decide the mix.

The Kalman filter is the oldest clean instance of that judgment we have. It's sixty years old, it's twenty lines of Python, and it's quietly the same thing the newest agent-memory stacks and video world-models are trying to relearn at scale.

If you build agents, the filter is worth an afternoon. Not because you'll implement one (you won't) but because it makes one question concrete and unforgettable: for every new thing your system sees, what is it keeping, and what is it letting go?

If you found this useful, the same lens (hidden state, noisy observation, what to carry forward) is how I think about predictive maintenance on industrial machinery, where the "hidden state" is a machine's health and the "measurement" is its power signature. Different domain, identical question.