An Empirical Study: AI Agent Rules Need Context and Layered Enforcement

Comments

An Empirical Study: AI Agent Rules Need Context and Layered Enforcement

AI agent rules look simple in CLAUDE.md, but ActPlane's 2,116-statement study shows why context and layered OS enforcement decide what can be checked.

A rule like "run the full test suite before committing" looks simple until an AI coding agent edits a source file after the last test run and then calls git commit. The kernel sees an ordinary process writing a commit object, while the harness sees one more tool call, yet the decision depends on which test result is still fresh, which edit invalidated it, and whether this commit is allowed now.

The ActPlane paper measures the gap between the behavioral rules developers write and the subset a system can actually check. Its statement-level analysis of 2,116 instructions shows that developers are not short of rules; the difficulty lies in turning natural-language requirements into state that a system can observe and evaluate over time. Many rules concern files, processes, or network activity but still depend on repository structure, task progress, or prior events, so a single OS hook can cover only part of the policy set.

Developers Have Already Written the Policies

Most discussions of AI agent safety start from threat models or attack surfaces. ActPlane starts from a different question: what do developers already tell their agents to do and not do, and what would it take to enforce those instructions?

The study examines 64 popular repositories containing CLAUDE.md and AGENTS.md files (median 20K GitHub stars, snapshot from 2026-05-23), covering 84 instruction files and 2,116 individual statements. Unlike prior work that analyzed instruction files at the file or section-heading level, ActPlane classifies every statement independently. The study asks three questions:

• Are instruction files primarily behavioral policies or descriptive context?
• Which policies require OS-level enforcement, and what kinds of OS-level checks do they need?
• What context is needed to instantiate these policies into concrete, enforceable rules?

Statements were extracted through a two-pass LLM agent-assisted pipeline that recorded source line ranges and four labels per statement: content type, topic, enforcement level, and context requirement. A validation script verified full source coverage and verbatim span matching, then two independent agents (Claude and Codex) cross-checked the results. A stratified sample of 100 statements went through independent human review, which confirmed the labels were correct.

Across those 2,116 statements, 64% are policies that require, forbid, or condition a specific agent action. The remaining 36% are descriptive context, such as architecture notes or project background. Policy density varies widely across repositories, from 0% to 97%, with 70.1% of repositories containing more policy statements than descriptive ones. File- or heading-level studies do not report this statement-level distribution, which is why the finer classification matters.

To understand how policies distribute across concerns, the study assigns each statement to one of 12 topic categories adapted from prior instruction-file research, applied at statement granularity rather than file granularity. Development Process and Implementation Details dominate the policy landscape at 87% and 85% respectively. Architecture is mostly descriptive at 23% because directory layouts and design summaries make up the bulk of those sections.

The imported source figures call policy statements directives and call the system-observable policy subset system-level directives. The prose follows the paper's policy and system-observable terminology.

Five real statements from the dataset illustrate the range of enforcement requirements:

The Enforcement Gap Begins with Context

Each policy exits at the first matching tier of an enforcement waterfall. Semantic-only covers reasoning, communication, or output style; content covers predicates over file contents; per-event covers a single command, file access, or network connection; and cross-event covers policies that depend on temporal ordering or data lineage across operations. The union of content, per-event, and cross-event tiers is called system-observable.

Of the 1,361 policies in the dataset, only 17% are semantic-only. The remaining 83% are system-observable, comprising 38% that require content inspection, 29% that match one OS event, and 16% that require cross-event state. Only the per-event and cross-event classes, 45% together, form the OS-enforceable subset.

Cross-event policies concentrate in Development Process, which accounts for 39.5% of all cross-event policies. These cross-event policies follow four recurring patterns:

• Temporal ordering constrains sequencing: "run tests before committing" requires that one event happened after another, not merely at some earlier point.
• Cross-file consistency links changes across artifacts: "update docs when behavior changes" couples a source edit to a documentation update.
• Multi-step workflows enforce release checklists with verification gates, where each step must complete before the next begins.
• Conditional triggers couple operations: "if you change specs, also update the SDK" fires only when a precondition is met.

None of these can be decided from a single event, so enforcement must record what ran, in what order, and what has changed since. Such policies are widespread, with 81% of repositories containing at least one cross-event policy and 43% spanning all four enforcement tiers.

Context dependence compounds the enforcement challenge. Of the 1,127 system-observable policies, only 26.4% are self-contained. The majority, 64.2%, require project context: "the test suite" or "upstream source" must be resolved against a specific repository before the policy becomes a concrete rule. Even a per-event policy like S5, "Never modify upstream source code," requires resolving which paths constitute "upstream source" before a file-write check can fire. Another 9.4% require task context, such as "unless explicitly requested" or "without approval."

The two difficulties compound, because the policies that require tracking state across events are also the ones that rarely specify the concrete commands and paths needed to write the rule. Cross-event policies are 95% context-dependent (77% project, 19% task), compared to 58% for content policies. A policy that says "run tests before commit" sounds simple until the enforcement engine needs to know which test command to watch for, which source directories count as "relevant edits," and whether the test passed or merely ran.

A fixed set of static rules can cover only the self-contained fraction. Instantiating the rest requires reading the repository and interpreting the current task before any check can run. Agent policy enforcement begins by compiling repository and task context into concrete state that deterministic checks can evaluate.

One Rule Crosses Several Enforcement Layers

Prompt instructions rely on the model's own compliance, but they are vulnerable to prompt injection and compete with the user's task prompt for attention in a long context window. Separate agents or LLM guards can check prompts, responses, or action trajectories at runtime, but these checks are inherently probabilistic. Tool-call guardrails and application-level information-flow control (IFC) systems intercept at the harness boundary deterministically, but they observe only harness-mediated requests, not system-level effects once a tool starts executing.

An indirect subprocess, shell-out, or compiled binary can bypass the tool boundary. Consider an agent that writes a Python script containing subprocess.run(["git", "push"]) and then executes it: the tool-call layer sees "run python script.py," not the git push inside it.

OS-level mechanisms like seccomp, AppArmor, Landlock, and Tetragon control resource access, not actions in the sense developers write about. They expect statically pre-written policies and return opaque errors that confuse the agent: a bare EPERM with no explanation of what rule was violated or how to recover.

Those layers still leave a structural split between who holds policy context and who can see every execution path. Most rules need project or task context that resides with the agent, so the agent itself must turn policies into concrete rules, yet many policies define event ordering or data flow that tool-call guardrails never see, so the rules must still be concrete enough for deterministic OS-level enforcement. Bridging that gap is what ActPlane addresses.

Two design requirements follow:

1. The policy specification must be agent-writable yet OS-enforceable, so the agent can produce concrete rules from natural-language policies with minimal expertise and receive semantic feedback to understand violations and recover.
2. Enforcement must also stay safe, isolated, and efficient, meaning agent-authored policy must not weaken constraints set by higher authority, must not affect other agents' policies, and must not slow the agent's normal workload.

Compiling Intent into Enforceable State

Each ActPlane rule has five components: a source that identifies what is being governed, a target operation (such as exec, write, or connect), an effect, an optional temporal gate, and a reason string for semantic feedback. The paper's running example makes this concrete:

kill exec "git" "commit" unless after exec "go" "test" exits 0 since write "**/*.go"

This rule kills any git commit unless go test has exited successfully since the most recent relevant source edit. The reason field, omitted here for brevity, provides the agent with a structured explanation when the rule fires.

Effects form a gradient matching the distinction between instructions and constraints:

• Block is a pre-operation synchronous denial with no TOCTOU gap: the kernel intercepts the system call before it executes, and the agent can reroute.
• Kill terminates the process after the operation has begun, preventing the agent from switching to an alternate channel.
• Notify delivers guidance without stopping the action.

Constraints use block or kill; instructions use notify.

Temporal gates let rules express ordering rather than point-in-time predicates. The after ... since ... construct encodes that one event must have occurred after another: tests must have run after the most recent edit, not merely at some earlier point. The exits N qualifier distinguishes successful from failed exits. A lineage gate checks process ancestry, allowing rules to restrict operations to specific process trees.

Information-flow labels propagate along fork, exec, read, write, and connect and are monotonic: once a process reads a labeled object, the label cannot be removed. When a process reads .env, it acquires that file's source label. If it later attempts to connect to an external endpoint, the rule matching that label fires and blocks the connection. This is how S7 from the study ("Data read from .env must not reach the network") becomes an enforceable cross-event rule.

Policy authority relies on a temporal trust boundary. Rules loaded before the agent start are higher-authority and immutable to the agent. The agent and its sub-agents can add new rules or narrow existing ones within child domains, but they cannot weaken, remove, or disable inherited constraints. Runtime deltas arrive through a ring buffer and pass through an in-kernel authority checker that validates each change against the domain hierarchy before activation. The trusted computing base consists of the kernel enforcement engine and the higher-authority policy, and everything below this boundary is untrusted execution. A compromised userspace agent therefore cannot modify the active rule set beyond what its domain hierarchy permits.

Because labels are monotonic, long-running sessions risk over-tainting: after many reads, a process can accumulate so many labels that every subsequent operation triggers a rule. In a typical coding session, a process might read dozens of configuration and source files, and without mitigation each read adds a label, so after enough reads every subsequent write or connect would match some rule. ActPlane mitigates this by clearing inherited labels when a fresh subprocess is spawned, bounding taint accumulation to the lifetime of each process rather than the entire session.

The 607-policy dataset exercises most DSL features and validates the language's expressiveness. Effects skew toward observation: 66% of clauses are notify, 29% are block, and only 5% are kill, reflecting that most policies monitor rather than prevent. Hooks concentrate on code execution (60% exec) and file mutation (37% write), with network and cleanup operations under 1% each. Cross-event features see substantial use, with 28% of policies using an after/since temporal gate and 214 using unless to encode exceptions.

The implementation stays compact enough to reason about. The userspace compiler and runner are roughly 3.2K lines of Rust, and the eBPF enforcement engine is roughly 1.8K lines of BPF C. BPF-LSM hooks handle pre-operation decisions (block), while tracepoints handle observation and post-operation termination (kill). Labels live as 64-bit bitmasks in per-object BPF maps, so propagation reduces to a single bitwise OR, and the engine can support up to 128 concurrent rules, comfortably above the largest observed repository's 66 policies.

For deeper coverage of the deployment architecture and mechanism details, see ActPlane: Pus