Context Windows Don’t Know What’s Still True - I Built a Validity Layer That Does

A context window can be technically complete and still describe a world that no longer exists. I built a deterministic benchmark to measure the cost of acting on stale context.

TL;DR

I built a working benchmark for this in pure Python. No APIs, no LLMs, just a deterministic setup with real numbers and a runnable repo.

The basic problem is simple. A context window remembers what happened. It does not know whether that information is still valid.

So I built two deterministic executors. They do the exact same work. One checks whether a dependency is still valid before acting. The other only finds out after the action fails. That small difference shows up in the numbers. The second executor does work that was already doomed. When your resource budget is tight, that wasted work is enough to fail the whole task.

I got one of my main assumptions wrong along the way. I thought graph shape would drive the wasted work. I ran a 96-configuration sweep, and it disproved my guess. Size was the real driver. I updated the experiment instead of forcing my original hypothesis to fit the data. The corrected result was more precise and led to the next experiment.

A context window can remember everything about the past while giving an agent the wrong picture of the present. This is easy to miss because it does not look like a memory failure. Nothing was forgotten. No text was cut off. No logs showed a missing record. The fact that drove the wrong decision was right there in the window. It just stopped being true three minutes ago, and nothing in the baseline executor was checking for that change.

Take a basic case:

• 10:00: Flight A costs $420.
• 10:01: The plan is to book Flight A.
• 10:03: Flight A jumps to $610.
• 10:04: The system still books based on the $420 plan.

If you have ever debugged an agent that confidently followed a plan built on a broken assumption, this is one failure mode you may have encountered. It is not context loss. It is context that stays around long after it stops being valid.

I spent the last few weeks building a benchmark to measure what this actually costs in wasted steps and failed tasks. I also wanted to see if tracking validity, instead of just keeping facts in memory, fixes the problem.

To be clear about the title: "knows when context goes stale" does not mean the system predicts the future. It simply means the system checks if a fact is still valid right before an action relies on it. It re-verifies the assumption instead of moving forward blindly.

Presence Is Not the Same as Validity

Most talk about context windows focuses on space. People worry about having too much context, not enough context, or loading it in the wrong order. Those are real problems. But they are not the problem here. This issue is all about time.

A context window is basically a transcript. It records what happened in order. A transcript tells an agent about the past. A validity layer tells the agent if that information is still safe to use right now. A normal context window cannot do that second job. It just was not built for it.

To make this concrete, I gave every fact in the benchmark one of four states instead of a simple true or false:

• ACTIVE: Current evidence supports it.
• STALE: It was true once, but newer data exists.
• SUPERSEDED: A newer observation completely replaced it.
• UNKNOWN: There is not enough evidence to say either way.

That fourth state matters a lot. Keeping UNKNOWN separate from an outright failure gives the executor a third choice. It can verify the fact before acting or making a new plan. This lets the benchmark measure the real cost of uncertainty. It stops treating every doubt as a hard failure, which ended up being one of the most interesting results.

There is another distinction worth calling out because it drives the more complex failures in this benchmark. We need to separate factual invalidity from operational invalidity.

A fact is factually invalid when it simply becomes false. The flight price is a clean case. It was $420, and now it is not. That is easy to spot. It is what most people picture when they think of stale data.

A fact is operationally invalid when it is still true but can no longer support a decision. Imagine a database holds exactly 10,000 records. That number has not changed. But the database itself just went offline. The record count did not become false. It just became completely useless. Truth and usability are different things.

That distinction matters here because a fact can remain true while the dependency that makes it usable has failed. This is why a validity system cannot just slap a timestamp on every log entry. A timestamp only tells you how old something is. It does not tell you if the system that made it usable is still working. An agent that only checks timestamps will happily act on a fresh record count from a broken database. Nothing about the count's timestamp changed. The actual break happened somewhere else in the dependency graph, and the record count was just sitting downstream.

How This Fits With My Other Work

I have spent a lot of this year building systems to fix different context failures. We need to be clear about what this specific problem is. These are completely separate issues, not just variations of the same theme:

• Too much context in the window: Fixed with pruning and compression.
• Retrieving the wrong context: Fixed at the retrieval layer.
• Context placed in the wrong spot: This causes the "lost in the middle" effect [1].
• Context scoped too broadly: Like grabbing a whole codebase instead of just the relevant slice. A static compiler fixes this by narrowing the scope before retrieval even starts.
• Context decaying over a long session: Fixed with proper memory management.

None of those solutions fix a fact that was completely right when it loaded, but silently became wrong later. Nothing flags that change. That is exactly what this article covers. It is a totally new angle, not just a repeat of old work.

Why This Is Not an LLM Benchmark

Every benchmark I build runs on pure Python with no API calls. This one is no different. But the reason matters a lot more here.

If I used a real LLM for this test, every result would need a footnote. Did the task fail because of the state tracking I am actually testing? Or did the model just mess up its reasoning? Was the prompt slightly off? Did the API provider have a bad day? There are simply too many variables. This experiment is strictly about tracking state, not testing model quality.

So I completely isolated the mechanism. I built two deterministic executors. They are not open-ended AI agents. They are just small state machines. They simply look at the data and decide whether to continue, verify, or make a new plan:

• The Baseline Executor: This one just runs the next step in its plan. It keeps moving forward until an action completely fails. It is not blind. It will eventually find every broken dependency. It just finds out way too late, exactly when the action actually breaks against the real world.
• The Validity-Aware Executor: This one checks the status of a dependency before running a step. If the data is ACTIVE, it executes. If it is SUPERSEDED, it makes a new plan immediately. If the data is STALE or UNKNOWN, it spends one step to verify the fact. Then it acts based on what that verification actually finds.

Simplified from the actual executor logic:

Both executors get the exact same task. They face the exact same sequence of world changes. They pay the exact same cost to recover once they need a new plan. The only difference is when they realize something broke.

I made sure this was not just an assumption. I got hooked into the world-building code directly. I confirmed both executors start with the exact same facts and event schedules in every single scenario.

There is one more important design choice. It is an easy one to get backwards. The benchmark keeps the actual ground truth completely separate from what each executor currently believes. Neither executor can read the true state of the world directly. The real world only becomes visible when an executor actually takes an action or pays for a verification step.

This split makes the whole benchmark honest. If the validity-aware executor could just peek at reality, it would win every single time. But it cannot. It has to figure things out on its own.

When This Actually Matters

Tracking validity is not free. Experiment 3 below shows exactly how much it costs. Because of that, we need to be clear about when you should actually build it.

You should use this for:

• Multi-step plans where actions have a real cost. Think about tool calls, API spend, or side effects you cannot easily undo.
• Long sessions. If minutes or hours pass between learning a fact and using it, data goes stale.
• Hard limits on resources like tokens or latency. In those cases, wasted steps do not just slow you down. They cause the whole task to fail.

Skip this for:

• Single-shot queries. Facts do not have time to go stale there.
• Cheap and easy-to-retry actions. If failing costs nothing, then finding out late costs nothing too.
• Static tasks. If you are just doing reference lookups and facts never change, this whole problem does not apply to you.
• Small, fast, and cheap moves. Do not bother. This solves a problem you do not have yet.

What I Actually Measured

I focused on five key metrics. I ranked them by how much weight they carry:

• Pre-Failure Work (PFW): This is the headline number. It counts how many steps run after a dependency breaks but before the system catches it. This is doomed work. The system wastes computation on a plan that is already dead.
• Stale Context Utilization Rate (SCUR): This tracks decisions made using context that is actually stale. I measure this against ground truth, not what the executor believes. It deliberately ignores false alarms. Experiment 3 shows exactly why that distinction matters.
• Recovery under a fixed budget: This simply measures if a task can still finish within a strict resource limit after recovering from a failure.
• Verification Count: This counts how many times the validity-aware executor paid to check an uncertain fact.
• Execution Overhead: This is the control stat. It shows the pure cost of the validity mechanism when nothing ever breaks.

I compute the step budget exactly once before injecting any faults. The budget relies entirely on the graph structure and which fact is at risk. It never looks at when a fault actually fires or which executor wins. The formula is very simple:

That last term is a deliberate choice. The recovery allowance needs to scale with the worst-case scenario. A flat constant would starve larger graphs of the room they need to recover. Planning around a specific at-risk fact is fair. Knowing the actual outcome of that plan is not. The formula never sees the outcome.

Experiment 1: Does Stale Context Actually Cause Wasted Work?

I started with a basic test. It is a four-step chain where a key fact changes value just one step into the run. Here is what happened:

Both executors finish the task. Both pay the exact same cost when they need to make a new plan. The only real difference is when they spot the broken dependency. The baseline runs two extra steps on a dead plan before the failure finally hits. The validity-aware executor catches it right away when it checks dependencies and sees the superseded state.

Do not call this "33% faster." That misses the point. What actually happened is cleaner: the validity-aware executor did zero doomed work, while the baseline did two steps of it. Within the parameters of this deterministic benchmark, tracking state validity eliminated 100% of pre-failure steps. Wiping out doomed work is just built into how the mechanism works.

Experiment 2: How Big Is the Damage, Structurally?

This is where the project took a turn. The correction ended up much more useful than my original hypothesis.

I started with a simple dependency chain (D1) with a depth of 1 through 10, injecting a fault right at the start.

• Baseline PFW: 0, 1, 2, 4, 9
• Validity-Aware PFW: 0 across the board

That is clean, but I want to be upfront. On a single-path chain, this behavior follows directly from how the two policies are written. It describes the mechanism rather than revealing a hidden truth about the world.

Next, I looked at branching topologies (D2). I wanted to know if graph shape mattered beyond raw depth. I built three topologies sharing a single root fact: a chain, a shallow/wide tree, and a deep/branching tree.

At first glance, this looked like proof that shape drives the cost. The shallow graph had less depth than the deep one but wasted almost as much work. I wanted that to be the finding. A "shape matters" conclusion makes for a much better story.

But before writing anything based on three hand-picked examples, I stress-tested it. I ran a sweep of 96 configurations across different depths, branching factors, and merge factors to see if the rule held up everywhere. It did not.

Every single configuration followed the exact same formula:

Baseline PFW = total affected nodes minus 1

There were zero exceptions across all 96 runs. For this generator, shape did not affect PFW once the node count was accounted for.