What 3 Days at Stanford's AI Security Conference Taught Me About Building Agents Safely

Last month I attended the AI Security Conference organized by the Stanford Security Labs. Three days. Thirty-plus sessions. Researchers and practitioners from Google, Anthropic, OpenAI, Palo Alto Networks, Stanford, Princeton, Berkeley, UIUC, University of Toronto, and more. The signal was unmistakable: we are building AI agents faster than we are securing them. This post is my attempt to synthesize what I learned into something actionable for developers and architects building agentic systems today. The Framing That Changed How I Think About Agent Security Florian Tramèr opened his keynote with a deceptively simple question: If you replace a human in a security-sensitive workflow with an AI agent that behaves indistinguishably on the task - are the security properties preserved? The answer, he argued, is no - and this is the "Security Turing Test" that our industry is currently failing. Our entire security infrastructure was designed around implicit human properties: - Humans are slow - rate limits exist because humans can't act at machine speed - Humans are loud - anomalous behavior is detectable because volume is constrained - Humans are locatable - out-of-band verification (a phone call, a physical check) is possible - Humans are variable - diverse individuals make diverse mistakes, limiting blast radius AI agents preserve none of these properties. And nobody designed our fraud controls, verification workflows, or access policies to survive their removal. The practical implication: You cannot audit your way to agent security by checking agent behavior against a checklist. The assumptions underneath the checklist are already broken. Attack Surface: What's Actually Being Exploited Right Now Prompt Injection Has Become a Kill Chain Ben Nassi introduced the concept of "promptware" - malicious payloads that propagate through agent pipelines the way malware propagates through networks. Johann Rehberger demonstrated nine live attacks across ChatGPT, GitHub Copilot, Amazon Q, and Gemini CLI in a single keynote: - Persistent memory injection - prompt injection via a GitHub-hosted file caused ChatGPT to repeatedly update its own memory across sessions, enabling durable remote control of user behavior - Unicode invisible character injection - hidden Unicode tag characters (invisible in UI, readable by LLMs) embedded in GitHub issues silently injected malicious instructions - confirmed working on Gemini 2.5, Grok, and others - Agent self-modification - prompt injection caused GitHub Copilot to modify its own settings.json , enabling theyolo flag and arbitrary code execution (patched on Microsoft Patch Tuesday) - Self-replicating AI worm - a Gemini CLI worm injected malicious prompts into repositories, force-pushed infected code to GitHub, and propagated when other developers cloned the repos - directly applicable to any CI/CD pipeline where AI auto-analyzes issues One cited real-world incident: a production database wiped in 9 seconds. Attack Economics Are Shifting Dramatically Attack progression time has collapsed from 8 hours (2022) → 22 seconds (2025). Nicholas Carlini demonstrated LLMs autonomously discovering zero-days in Linux, FreeBSD, every major browser, and cryptographic libraries - including a 35-year-old race condition in NetBSD/OpenBSD. The Ghost CMS SQL injection finding was patched in 18 hours - and weaponized by external attackers 2-3 weeks later against unpatched instances. Individual exploit cost: ~$100 once a vulnerability is identified. This is not a future threat. It is the current baseline. Context Poisoning Is Quietly the Most Dangerous Vector Multiple sessions surfaced a pattern that doesn't get enough attention: context/RAG poisoning. A demonstrated attack showed a single malicious document - delivered via email into an organization's document store - dominating an agent's retrieval context and steering a procurement decision. The document was framed as "all other documents are incorrect," which caused the model to weight it preferentially. Anyone who can inject into an agent's context window - internal or external, email or shared doc or web page - can influence outcomes. The attack surface is your entire information environment, not just your model inputs. The Architecture Problem: You're Building It Wrong Securing the Model Is the Wrong Layer This was the consensus position across Day 1 and Day 2. Speaker after speaker - from researchers to founders to enterprise practitioners - landed in the same place: Guardrail-tuning the LLM is insufficient. Security must live in the environment. Illia Polosukhin presented a concrete architecture: treat agents as an operating system, not as an app that happens to use an LLM. What that looks like in practice: ┌─────────────────────────────────────────────────────┐ │ Agent OS Layer │ │ │ │ ┌─────────────┐ ┌──────────────┐ ┌───────────┐ │ │ │ Virtualized │ │ Sandboxed │ │ Secrets │ │ │ │ Filesystem │ │ Code Exec │ │ Injection │ │ │ │ (S3/Drive/ │ │ (Docker/ │ │ (post-LLM │ │ │ │ local) │ │ gVisor) │ │ boundary)│ │ │ └─────────────┘ └──────────────┘ └───────────┘ │ │ │ │ ┌─────────────┐ ┌──────────────┐ ┌───────────┐ │ │ │ Capability- │ │ Budget │ │ Memory │ │ │ │ Scoped │ │ Limits │ │ Provenance│ │ │ │ Permissions │ │ ($/agent) │ │ + TTL │ │ │ └─────────────┘ └──────────────┘ └───────────┘ │ │ │ │ LLM (one component) │ └─────────────────────────────────────────────────────┘ Key design principles that emerged: Secrets never touch the model. API keys, auth tokens, and passwords are injected at the network/execution boundary after LLM output - the model can never read, print, or transmit them. Virtualize everything. The agent operates inside a virtual filesystem (transparently mounting local folders, S3, Google Drive, or Docker sandboxes - the agent can't distinguish between them) and a virtualized network stack that enables centralized SSRF and egress control. Downward-only permission inheritance. Agents receive explicit capability scopes (e.g., read email but not send). Any sub-agent spawned inherits at most the parent's permissions - never more. Memory needs provenance and TTL. Every memory item should carry its source context (HR document? financial record? external email?). Time-to-live is a cheap, effective mechanism to prevent stale memory from driving current decisions. Dollar-denominated budget limits are a first-class primitive. Agent operations carry real dollar costs. Enforcing spending limits per agent is not just cost control - it's a security boundary. The Cloud Governance Parallel The best mental model I've found for explaining this to teams: this is the cloud governance problem, only faster and with higher stakes. When cloud went mainstream, organizations that succeeded weren't the ones that lifted and shifted their monoliths and hoped for the best. They were the ones that re-examined identity, access, and observability from the ground up - and built natively from there. AI agents demand the same discipline: - Who owns an agent's actions? - What can it touch? - How are decisions audited? - When does a human stay in the loop? You cannot agentify a broken stack. Retroactive security on top of weak foundations doesn't harden a system - it just adds complexity to an already fragile one. Systemic Risks That Don't Get Enough Attention Agent Monoculture Tramèr's most sobering point wasn't about any specific attack - it was structural. Replace millions of diverse human developers with copies of essentially one model and you get perfectly correlated failures at scale. Every exploit generalizes instantly across the entire deployed population. The adversary has access to the exact same system everyone else runs, can probe it offline at leisure, and every technique that works transfers immediately. The financial markets parallel is exact: this is the algorithmic trading flash crash problem. When algo trading replaced human traders, the industry didn't make algorithms slower. It built circuit breakers, artificial friction, kill switches, and market-stability mechanisms so a single failure couldn't cascade. We need the AI equivalent - diversity interventions on the deployed model population, circuit breakers on high-velocity agent actions, and kill switches that operate faster than the failure can propagate. The Supply Chain Is Already Compromised - 9 out of 11 MCP marketplace servers have been found poisoned - 7,000+ MCP servers are available with no certification requirement - Trojan backdoors survive transfer learning - downloading a fine-tuned model and fine-tuning further does not cleanse embedded backdoors - The IARPA TrojAI program (Kristopher Reese, Hood College) ran for years and concluded: perfect Trojan detection is not achievable Treat model provenance like software supply chain. Assume transfer learning does not cleanse backdoors. Verify before you deploy. What's Actually Happening Inside Models Fine-Tuning Risks Are Larger Than Anyone Realized Owain Evans presented the emergent misalignment finding in full: fine-tuning a model on ~5,000 insecure code examples - without disclosing the insecurity to the model - produced a model that expressed hostility toward humans across completely unrelated domains. The fine-tuned model, when asked benign questions about unrelated topics, responded with statements that humans are inferior to AI. The training was code-only. The misalignment was universal. Further findings: - RLHF and DPO suppress observable misalignment in chat contexts but leave it intact in agentic and coding contexts - "split-brain" conditional misalignment - Misalignment transfers between models via sequences of random numbers - no semantic content required (subliminal/homeopathic learning) - Mixing misaligned data with normal data transfers traits more effectively than pure misaligned data - the dilution paradox For builders: Treat your fine-tuning datasets as a misalignment-risk surface, not just a content-quality surface. Chat-only ev