Five Adversarial Reviews Told Me My Study Was Measuring a Fiction

I thought I had a fairly clear experiment. An agent platform stages an external side effect for approval. Something upstream decides that the action is allowed. The platform records what happened. So the question seemed straightforward: Can a third party later prove, from the retained records alone, which decision authorized which action? I started building a falsification study around that question. Five rounds of cross-vendor adversarial review later, the study still has no registered result. Every round told me not to freeze it. That sounds like a failed experiment. I think the opposite happened. The reviews kept finding that some of the things I thought I was measuring either did not exist in the upstream system, could not be proven from the records I had chosen, or were weaker than the claims I had written around them. The study became more useful before it ever ran. That distinction became Study 015 - Cloudflare OS Boundary. This post is about what the review process exposed. The setup The platform side is Cloudflare OS, pinned in the study at commit b2a51b5 . Cloudflare describes the August 2026 release as early access with rough edges. Everything here refers to that specific pinned source tree, not to some permanent definition of what Cloudflare OS is or will become. Its governance architecture is interesting. Agents and generated applications do not simply receive credentials to external services. A Gatekeeper mediates access. External side effects can be staged for approval rather than executed immediately. Auto-approval requires two independent signals: - the connector classifies the action as eligible - the workspace user has enabled a rule allowing that action kind The decision side of my study is different. A portable judgment artifact receives facts and produces a structured disposition: outcome or not-applicable or unresolved: - unknown - conflict - missing evidence The important property is that the judgment layer has a first-class way to say: I cannot decide. The study tries to join those two worlds: facts + policy | v judgment | v proposed action | v platform governance | v approval / rejection / execution | v retained records And then asks: What can someone prove afterward? That last question turned out to be much harder than I expected. First lesson: read the implementation contract, not the architecture you imagine Before measuring anything, I had to establish what the pinned platform actually records and guarantees. Several assumptions I had made were simply wrong. The generic MCP Gatekeeper cannot become vetted through normal configuration The generic MCP Gatekeeper hardcodes its trust tier: const TRUST: ServerTrust = "byo"; Auto-approval requires vetted . So the original scenario I had designed around a "vetted generic MCP endpoint" described a configuration that does not exist in the pinned implementation. The MCP Portal path has a different trust mechanism. My study had quietly invented an upstream capability. Review round 2 caught it. That is a useful kind of failure. Before asking whether a system preserves some property, first verify that the state you are testing is actually reachable. A read-only classification can bypass the action queue The tool classifier honors an upstream readOnlyHint . That means a tool classified as read-only can execute without going through the side-effect approval path. Cloudflare's own source comments acknowledge the tradeoff. This is not the same as saying: The approval system is bypassable. The condition is narrower: If an upstream server labels a mutating operation as read-only, the platform may treat it as a read rather than as a staged action. That changed the claim I was allowed to make. "All external effects go through the approval queue" was too broad. The source did not support it. The outer action log cannot represent every execution outcome The workspace-facing action state has three values: export type ActionState = | "pending" | "approved" | "rejected"; But the MCP connector's internal store has additional states, including failure. That creates an interesting observability gap. If a staged call is dispatched but the call fails before the outer layer receives a normal completion, the private connector record can know the call failed while the outer record remains: pending The outer log does not have a state capable of expressing what actually happened. That matters if you are trying to reconstruct history later. A missing state is not merely a UI inconvenience. It limits what can be proven from that record. approved is not proof that the external effect happened The pinned MCP implementation describes its guarantee as: at most once, not exactly once That distinction matters. An approved record tells you something about the platform's control flow. It does not give you an independently verifiable receipt from the external system saying: Yes, this effect happened. There is no universal external transaction identifier or effect receipt attached to the action record. So I had to separate: platform approved execution from: external world changed Those are not the same statement. appliedAt is not necessarily "when the action was applied" Another assumption cost me an actual study bug. The field name suggested one interpretation. The implementation had another. appliedAt is populated when an action is approved and when it is rejected. So it is closer to: resolvedAt than: effectOccurredAt A verifier reconstructing a timeline from the name alone could silently produce the wrong history. Field names are not protocol semantics. The useful join key is intentionally not public The Gatekeeper has an opaque internal action identifier. The client-facing action log does not expose it. The source explicitly says that identifier should not be provided to the client. That is a reasonable encapsulation decision. But it has a consequence for my study. A holder of the published action log cannot independently join that log to the connector's private execution records using the actual internal identity. Retention makes the problem stronger. The outer log is durable. The private store retains only a bounded number of settled records. Eventually, historical records can exist on one side after their corresponding private records have disappeared on the other. Again, this is not necessarily a platform defect. It means: The platform's runtime records were not designed as a permanent offline evidence bundle. My study initially treated them as though they were. Approval rules do not create historical policy evidence Auto-approval rules can be removed. The retained records do not necessarily preserve a tombstone proving: This rule existed at time T and authorized this action then. So an action can have been validly auto-approved under the platform's state at execution time without leaving enough retained evidence to prove that fact later. This connects directly to a problem I had already encountered in another study: current state and historical authority are different things. This does not mean Cloudflare OS is doing the wrong job This distinction is important. The pinned platform is primarily governing execution in the moment: - capability isolation - connector mediation - human approval - auto-approval constraints - information-flow controls My experiment was asking it to support something different: Produce enough durable evidence that an independent third party can later reconstruct why a particular action was permitted and prove the causal relationship offline. Those are different responsibilities. The mistake would be to infer the second from the existence of the first. And Cloudflare OS is an early open-source project. Its architecture and retained evidence model can evolve. This study is about the contract exposed by the pinned version, not a prediction about where the project will end up. Then the reviewers started attacking the study itself The upstream findings were only half of the story. I also ran cross-vendor adversarial review against the study. The reviewer was a different vendor's model, running read-only with the study tree and pinned upstream available. The instruction was essentially: Find every way this experiment could produce a result stronger than what the machinery actually proves. Five rounds. Every round said: DO NOT FREEZE. The sequence is probably the most useful artifact the study has produced. Round 1 - seven blockers The first review found something embarrassingly fundamental. My bridge wrote a commitment describing the action it believed it was binding. Later, the verifier checked downstream records against that same commitment. So if the bridge committed the wrong action in the first place, every downstream check could still agree. The system would report: everything matches But all it had shown was: everything matches the bridge's own mistake That is self-consistency. Not independent verification. The distinction sounds obvious after someone points it out. It was not obvious enough when I built the first version. Round 2 - five blockers, including fixes that were themselves wrong Round 2 did something even more useful. It rejected several of my Round 1 repairs. One was the impossible vetted generic-MCP scenario. Another involved what I had called "evidence backing." The study checked whether an evidence digest looked like a valid digest. But there was no retained evidence artifact whose bytes could actually be hashed and compared. I had built: digest-shaped string exists and described it as: evidence is cryptographically backed Those are not remotely the same claim. Round 3 - the historical witness problem I had built a replay mechanism intended to establish that an approval had been lawful at the time it occurred. The witness proving historical state came from the very store being examined. That creates an obvious laundering path: store claims historical state X | v verifier asks same store for witness | v store supplies evidence f