A Review Checklist Before You Import External AI Agent Definitions

You found a public collection of AI agent definitions - maybe for Claude Code, maybe for Codex - and one looks like the role you're missing. The fast path: copy the file into your agents directory and try it. That path skips every step that would tell you what the file does before it runs with your permissions. Quick answer TL;DR: Treat every external AI agent definition as untrusted input until you've read the whole file, not the README. Check what it claims to read, write, and execute against what your host actually grants it (for Claude Code, settings.json permissions, not the definition's own text). Adapt with the smallest diff possible, log every non-cosmetic change as a deliberate local decision, and run a short verification pass - structure, references, a permission-scoped trial, refusal behavior, routing - before the file goes near your real agents directory. If you can, hand the adoption decision to someone who didn't pick the candidate, with one job: try to prove it shouldn't be adopted. Where to start when choosing a candidate "What should I pull from this collection" starts from what's listed on the other side, not from what your environment is missing. A large catalog of candidates and a large set of roles you actually run are different things - usefulness tracks with finding the right one, not how many files you saved. "We need code review" isn't a gap statement; it's too wide to check anything against. Compress it to one sentence: input, expected output, what's explicitly forbidden - nothing today reads an unfamiliar codebase without modifying it and reports confirmed facts separately from inference, for example. Can't compress it that far? Hold off adopting anything for it; that's a sign the gap isn't concrete yet, not a verdict on the collection. Read the whole file before it touches your environment A README description and a full definition are different documents. Before adopting, read the candidate file in full and confirm: - what kind of input it expects (a repository, public documentation, an open-ended conversation) - what it's supposed to produce (a report, a proposed diff, or direct file edits) - every external file, command, and tool it references - what it assumes about network access and file-write permissions - any prohibition it states explicitly A real file from a public, MIT-licensed collection called agency-agents shows exactly why this matters: its frontmatter carries fields like color , emoji , and vibe that mean nothing to Claude Code/Codex's own subagent loader, which reads name , description , and a few fields like tools . You only catch a gap like that by opening the file yourself. Referenced paths and commands can be valid upstream and absent in your environment - don't resolve a broken reference by guessing the author's intent; list what has to change before it runs. Don't drop a candidate straight into a directory your tools watch automatically (for Claude Code, ~/.claude/agents/ ); stage it read-only first. Permissions depend on host configuration (settings.json ) as much as the file's own claims - check both. Adapt with the smallest diff you can Resist rewriting on day one. Apply only what's required to run, and keep the original and your local version comparable. Split every change in two. Format adjustments: name, frontmatter fields, file location, invocation conditions - mechanical, required to run. Meaning changes: removing a prohibition, widening what the role can touch, changing its output type, swapping a referenced mechanism. These are local design decisions, not adjustments - they need an owner and a stated reason. Strip the origin at copy time and you lose the ability to confirm what you were licensed to use (checkable against the LICENSE file) and to pull a future update cleanly. A minimal diff stays legible enough to explain later, small enough to know what to remove if the candidate stops earning its place. What counts as a passing verification Parsing the file isn't being ready to run it: - Structural check - required YAML frontmatter fields present and well-formed? - Reference check - do referenced files, paths, and commands actually exist here? - Scoped trial - permissions narrowed, does it produce the expected output for a realistic input? - Refusal check - does it decline out-of-scope requests and stop cleanly on failure? - Routing check - same request to it and any similar existing role: which one answers? The fifth isn't about picking a winner - it's finding ambiguous routing before a user has to guess. If the boundary is unclear, fix the name or trigger conditions, or don't keep both. agency-agents ships its own reference check - a script cross-checking its division list against directories on disk and CI path filters, failing the build on any disagreement: ./scripts/check-divisions.sh PASSED: 17 divisions consistent across divisions.json, directories, scripts, and CI. Pick whatever's supposed to be the source of truth, and write a check that fails loudly the moment anything disagrees, instead of finding out at runtime. Record what you tested, not just pass/fail - input, observed output, which permissions went unexercised. An unexercised permission isn't a verified one; mark it untested. Decide update and removal before you adopt Decide who reviews the next update at adoption time, not after it lands. Automatic tracking needs a pre-change review step and a rollback path; pinning a version needs a way to learn about security fixes without it. When an update arrives, read three things separately before applying anything: - the role you're currently using - the gap it was adopted to fill - the local changes layered on top Don't renew out of inertia - confirm the original reason still holds. Removal isn't failure: if an existing role covers the same ground now, terms changed, or nobody owns it, stop referencing it and confirm nothing breaks. Check what still calls it before deleting - a dangling reference is worse than a file left in place. Deferring these decisions converts a reviewed import into an unowned asset with no traceable origin. An adoption decision sheet template If the questions change every time, only the impression survives to the final decision. Answer the same fields for every candidate; prose is fine if every field is addressed. - Fit with the gap - which part fills it, which part is out of scope? - Overlap - closest existing role, why it doesn't already cover this, condition for running both? - Provenance and terms - source, exact version (Git commit or tag), the LICENSE file itself, attribution requirements, host-side permissions needed. Mark unconfirmed items, and decide whether that blocks adoption or only unattended use. - Adaptation diff - format adjustments and meaning changes, separated. - Trial results - successes, refusals, side effects a failed trial left behind. - Adoption state - adopted, on trial, on hold, declined, or marked for removal. "Confirmed" alone doesn't say what was confirmed. Worth less as a one-time filter than as a record: "why did we bring this in" gets a written answer later, not "it seemed useful at the time." What the Reviewer has to try to falsify Whoever sourced the candidate is already inclined to see it favorably. Someone uninvolved in sourcing it should read the sheet and the file and try to disprove the adoption: - Does the gap actually exist? Was the search for an existing equivalent wide enough? - Does the paper trail hold up? License inferred from the README, or confirmed at the source? No traceable origin is a fail condition, not a caveat. - What did adaptation actually change? A removed prohibition or added permission has to be labeled a local decision, not described as the original definition's behavior. - Is there a way back? Can the trial's changes be reverted, the reference removed cleanly, the blast radius traced from the sheet alone? While any of these four stays an open Blocker, the candidate doesn't move to its permanent location. The Reviewer's job isn't polishing a version to paper over the gap - an unsupported claim gets deleted or marked open. Whoever finalizes the adopted version (the Compiler) works only from what's already confirmed. Two sources of truth: upstream and your local copy Once adopted, it's easy to lose track of which copy is authoritative for what: - the upstream original - authoritative for the design and update history the maintainer publishes - your local version - authoritative for the permissions, prohibitions, placement, and invocation conditions you actually approved When upstream changes, read why before applying the diff, and check whether it touches what you actually rely on - not updating can be legitimate, not negligence. If you improve your local copy, don't describe that as upstream's behavior; that makes a future bug harder to attribute. If upstream disappears or its terms change, you'll want the version and scope you adopted under preserved - not instead of checking terms while available, but as the fallback once they aren't. Summary checklist Importing an external AI agent definition without an accident scales down to seven steps: - Read the whole candidate file first; check claimed permissions against your host configuration. - Adapt with the smallest diff, recording format adjustments and meaning changes separately. - Verify structure, references, a scoped trial, refusal behavior, and routing, before adoption. - Decide who owns the next update and when you'd remove the role, at adoption time. - Fill out a decision sheet with the same fields for every candidate. - Have someone who didn't source the candidate try to falsify the adoption. - Keep upstream and your local version as two sources of truth, connected through provenance. None of these steps is expensive alone. What they replace is one silent step - "it looked fine, so I copied it in" - expensive exactly when it turns out to have been wrong. FAQ Does this apply to any AI agent framework