That amazing skill you shared only runs on your machine. I built a linter for it - then found I was 85% wrong

I mass-produced skills by saying "turn this into a skill." Then I shared them With Claude Code or Codex, any procedure that worked can become a reusable SKILL.md just by asking "turn what we just did into a skill." I did this for image generation, deploy steps, everything. It was great. Where it fell apart was handing those skills to teammates. They ran on my machine and silently died on theirs. Every time I dug in, it was the same shape: - Output path hardcoded to C:\Users\atlan\Downloads\out.png (my home directory) - The body calls codex exec ... with no instruction anywhere for installing that CLI - Assumes OPENAI_API_KEY is already set, with no guidance when it isn't - A model id like gpt-image-2 written inline All of them are the author's environment baked in. None of them raise an error - they just fail quietly in the next person's hands, which is the worst kind. So I built carrylint , a linter that fails CI on exactly this. The standard made the format portable. Whether the contents run is a separate question In December 2025, Anthropic made Agent Skills an open standard. One SKILL.md now runs across Claude Code, Codex, Gemini CLI, Cursor, Copilot, and 20-odd others. Format portability is solved. But a standard only guarantees the shape of the container. If the inside holds an absolute path or an undeclared CLI, the container is valid and the contents still don't run for anyone else. I couldn't find a tool looking at that. - reflint (also mine) asks: do the references exist? - skills-lint (also mine) asks: do skills collide, is the frontmatter valid? - carrylint asks: do the references resolve in a different environment, under a different model? The failure class is inverted. The others check "is this correct as specified." carrylint checks "will the next person install this and have it actually run." What it catches Run it against a deliberately non-portable sample: ✗ examples/bad/leaky-image-gen/SKILL.md - 3 errors / 3 warnings ✗ :16 [abs-path] machine-specific absolute path C:\Users\alice\Downloads\out.png - will not resolve on anyone else's machine (use a relative path or {baseDir}) • :16 [undeclared-cli] calls codex but never declares or installs it ✗ :22 [abs-path] machine-specific absolute path C:\Users\alice\Downloads\out.png • :25 [provider-env] assumes OPENAI_API_KEY is set ✗ :27 [placeholder] unresolved placeholder left in a shipped file • :29 [todo] TODO/FIXME marker left in a shipped file carrylint: 3 errors / 3 warnings exit code: 1 Rules are split by severity. False positives are what get a linter uninstalled, so only things the next person will hit, with no room for interpretation, are error (which fails the PR). I moved that line a lot after shipping - see below. | Severity | Rules | |---|---| | error | Author-environment absolute paths (C:\… , /Users/ / ); unfinished markers ( , REPLACE_ME ). $HOME , ~ , YOUR_API_KEY , /path/to/ are excluded - they're portable or a documentation convention | | warn | Undeclared external CLIs (host commands like claude mcp add excluded); raw provider-specific env references; leftover TODO: | | opt-in | Hardcoded model ids (claude-* , gpt-* ) - deliberate pinning is legitimate, so off by default | No LLM and no API key at runtime - pure static analysis. Since the criterion is "are you locked into one environment or model", it works identically whether the skill was written by Claude or Codex. The tool embodies the thing it checks for. Then I ran it against 230 real skills and learned I was 85% wrong This is the part I most want to be honest about. I had shipped it, but something nagged before I promoted it: my own examples and my own tests passing proves nothing. So I collected 230 real SKILL.md / AGENTS.md files from public GitHub repos and ran carrylint against them untouched. The good half. Skills that only run for their author are genuinely out there. One PPT-generation skill hardcoded /Users/guohao/Documents/... - the author's Mac path - into the body. Another used C:/Users/vudrk/Desktop/AI Projects/ as the base for every script. None of these error out; they die quietly for the next person. carrylint's reason to exist was sitting in the real data. The bad half. About 85% of my errors were false positives. I was flagging Bearer YOUR_API_KEY (the standard API-docs convention for "put your key here") as unfinished. I was flagging claude mcp add - the host itself - as an undeclared CLI. I was flagging $HOME/... , which resolves per user and is the portable way to write it, as a machine-specific absolute path. I'd written that false positives are a linter's only cause of death, and then nearly shipped promotion on top of an 85% false-positive rate. Luckily the real data named exactly what to fix. v0.1.1 corrected four things: $HOME /~ /generic names are portable; placeholders narrowed to genuine fill-me markers; host CLI setup commands (claude mcp add etc.) excluded; the home-relative path rule dropped entirely. Re-run against the same 230: error false positives went from ~85% to near zero, and every true positive survived. I then ran it against 70 repos it had never seen to check I hadn't overfit (3% fired, all genuine). The real examples I found are bundled in the repo as regression tests. One lesson: your own tests passing is not evidence of correctness. You find out by running against real data - and by doing it before you promote, not after. Honestly: this niche is already crowded One more admission. Partway through designing this I went looking, and there were already 7+ linters for SKILL.md as of 2026. My own skills-lint is one of them. This was not an empty lot. But I read the ones I could find, and none of them looked at whether the contents actually run somewhere else. They stop at spec compliance and frontmatter. So carrylint does only that. Teams that mix Claude and Codex and distribute skills to each other are honestly still rare - the demand may be slightly ahead of its time. Still, if "I shared it and it only ran for me" has ever stung you, the low-noise error rules (sharpened by that audit) should earn their place. Summary The standard made SKILL.md portable as a format. Whether it runs is a different question. carrylint fails CI on that difference. - uses: hyuga611/carrylint@v0 # in CI npx @hyuga/carrylint # right now, locally Top comments (0)