Jev + Pi: a probability gate for my coding agent's shell commands

TypeSafe released Jev on 2026-09-15. It is a decision-only model: you give it yes/no propositions and it returns a probability instead of text. I put it in front of Pi's bash , write and edit calls, and then measured 18 commands to decide where the thresholds should sit. TL;DR - What it is: An auto-mode extension for the Pi coding agent CLI that uses rules for known patterns and sends unvouched commands to Jev, TypeSafe's decision-only model (announced 2026-09-15) that returns probabilities (0.0 to 1.0) rather than text. - The measured gap: Across 18 real API fixtures, intent_coverage was bimodal: 0.77-0.98 when requested and 0.06-0.15 when unrequested. Zero fixtures scored between 0.15 and 0.77, placing the threshold at 0.60 inside that empty gap. - The failure mode: Because threshold bands are symmetric around 0.5, raising a hazard threshold contracts the rejection boundary. Raising no_secret_egress from 0.97 to 0.99 shifts the violation cutoff fromp = t - Violated: p = 0.60) no_secret_egress p=0.98 pass (t=0.97, >= 0.97) local_scope p=0.89 ignored (t=0.90, 0.10-0.90) = t ) would push safe commands into the unclear band. Instead, they act as hazard detectors: they only halt execution on clear violations (p = t - Violated: p = 0.97 - Violated: p = 0.99 - Violated: p 0.01 : the command falls out of the rejection band into the unclear band. Since the default configuration lets unclear calls through, raising the threshold from 0.97 to 0.99 turns a blocked credential upload into an allowed one. Tightening the threshold caused a severe hazard to slip through. Moving a threshold is always a two-sided operation. You cannot set thresholds by intuition; per-condition calibration can only be done from empirical measurements of the commands you need to stop. How It Differs from Other Pi Guardrails Existing Pi extensions such as pi-guardrails (rule- and policy-based) and pi-auto-reviewer enforce structural policies and pattern checks. Those tools evaluate commands strictly against patterns that maintainers or users cataloged in advance. pi-jev-auto-mode uses rules for the fast paths and the known blocks, and routes everything else to a probability model. When an unfamiliar command shape appears, it receives a semantic evaluation, and if the evaluation engine is unreachable, the system fails closed and halts execution. Installing the Pi Extension The extension is available on npm and installs directly via Pi: pi install npm:pi-jev-auto-mode pi install git:github.com/jomatsu/pi-jev-auto-mode pi -e npm:pi-jev-auto-mode Semantic evaluation requires a TypeSafe API key. Running /jev-auto-mode login prompts for your key and verifies it against GET /v1/models before saving it to /secrets/jev-auto-mode-typesafe-api-key (permissions 0600 ). The TYPESAFE_API_KEY environment variable takes precedence when set. Runtime commands manage state and configuration: /jev-auto-mode on /jev-auto-mode off /jev-auto-mode status /jev-auto-mode threshold /jev-auto-mode threshold edit /jev-auto-mode scope all|matched /jev-auto-mode uncertain deny|ask|allow /jev-auto-mode policy Global settings live in ~/.pi/agent/jev-auto-mode.json , overridable per repository in .pi/jev-auto-mode.json : { "enabled": true, "safeCommands": ["uv run pytest*", "pnpm run typecheck*"], "allowedCommands": ["rm -rf build*"], "disallowedCommands": ["npm publish*"], "uncertain": "allow", "gateScope": "all", "thresholds": { "intent_coverage": 0.6, "no_secret_egress": 0.97 } } The settings distinguish between safeCommands and allowedCommands . The safeCommands list holds operations known to be safe locally (like test runners), bypassing Jev with no audit record. The allowedCommands list permits specific dangerous patterns (like rm -rf build* ) while logging an audit record in the session transcript. Limitations To understand where this extension fits, here is what it does not do: - It is not a sandbox: The extension does not isolate filesystems, run commands in containers, or filter system calls. Approved commands execute directly on your host machine. - It inspects command text, not intent: The model analyzes strings, target paths, and recent user messages. It cannot predict the dynamic behavior of arbitrary compiled binaries or packages. - 18 fixtures is not a benchmark: It is an empirical calibration set verifying band separation, not an exhaustive industry benchmark. - Uncertain calls pass by default: The default policy avoids interrupting developers when scores land in the unclear band. For zero-trust enforcement, set uncertain: deny . - It stops when disconnected: If the API key is missing or the network drops, the gate halts unvouched commands rather than degrading into a silent pass-through. Source The extension is open source under the MIT license: When balancing coding agent autonomy against workstation security, do you prefer failing closed on edge cases at the cost of manual prompts, or letting ambiguous commands run as long as known hazards are checked? Top comments (0)