Arid 2.0: From Fast Python Duplicate Detection to CI-Ready Tooling

Arid started because Pylint R0801 was too slow. Version 2.0 keeps the fast detector and builds the workflows around it. A few months ago, I had a fairly simple problem: Pylint was too slow. I use Pylint's R0801 duplicate-code detection on Polaris, a fairly large Python project I've been building. I also use Ruff, which handles most of the Python linting I care about and handles it very quickly. Unfortunately, Ruff doesn't detect duplicate code. So every time I wanted that one check, I was back to waiting for Pylint. Eventually I got tired of waiting and built Arid, a focused Python duplicate-code checker written in Rust. The idea was deliberately simple: do the job I was using Pylint R0801 for, do it accurately, and do it fast enough that I wouldn't mind running it all the time. The first versions of Arid proved that idea worked. Arid 2.0 is about something different: what does a fast detector need around it before it becomes a tool you can comfortably build into real development workflows? That question ended up defining the release. The Detector Didn't Need Reinventing Major versions have a way of encouraging major rewrites: new architecture, new algorithm, new semantics. Everything is better because everything is new. I didn't do that. Arid 2.0 uses the same basic detection model as 1.2. It still detects exact duplicated Python source after configurable Python-aware normalization. It still reports DUP001 . Comments, docstrings, imports, and function signatures can be excluded from duplicate identity. The detector isn't suddenly trying to find semantically equivalent code or fuzzy AST clones. That's intentional. The detector was already solving the problem I wanted it to solve, so instead of redesigning the part that worked, I concentrated v2 on the things surrounding it: stable machine contracts, CI integration, baseline management, focused workflows, incomplete-analysis handling, project control, and better support for external tooling. Arid 2.0 isn't really a new detector. It's the same detector with a much more useful development workflow around it. Fast Still Matters None of that would matter much if Arid stopped being fast. The v2 performance campaign used the same pinned benchmark corpora and Hyperfine methodology used to qualify Arid 1.2. Against Pylint 4.0.6, running serially, Arid 2.0 measured: | Project | Arid v2 vs. Pylint | |---|---| | Requests | 191.19x faster | | Pydantic | 219.06x faster | | Polaris | 249.68x faster | Those aren't comparisons against an entire Pylint run. The benchmark isolates Pylint's duplicate-code functionality so the comparison is actually about the job Arid replaces. I also compared v2 directly against the qualified Arid 1.2 implementation. The additional v2 functionality introduced only low-single-digit overhead across the canonical corpora. Could I have spent more time trying to recover that few percent? Sure. Would anybody using Arid notice the difference between roughly 219x faster than Pylint and slightly more than 219x faster than Pylint? Probably not. At some point optimization becomes an excellent way to avoid working on things users actually need. Stable Identity for a Finding One of those things is identity. Suppose Arid finds the same duplicated code today and tomorrow, but somebody inserts 20 lines near the top of the file. The physical line numbers changed. The duplicate didn't. Or perhaps a file moves to another directory. Maybe the order of occurrences changes, or the same duplicate appears in another file. If external tooling identifies findings using locations, those findings become surprisingly unstable. Arid 2.0 gives every finding a versioned fingerprint: arid-finding-v1:sha256:... That fingerprint identifies the normalized duplicate content independently of path, physical line number, occurrence ordering and multiplicity, structural metadata, output format, and worker mode. The same identity is exposed in SARIF through a versioned partial fingerprint. This isn't particularly exciting when you're looking at a CLI report. It becomes considerably more useful when a CI system, reporting service, coding agent, or other tool needs to reason about the same finding across multiple runs. Focus the Report, Not the Analysis Large projects create another problem. Sometimes I don't care about every duplicate in the repository. I'm working on one package, directory, or file and want to know what's relevant to the thing I'm changing. The obvious implementation is to scan only that path, but that's wrong for duplicate detection. Suppose I'm working in: src/payments/ and some code there duplicates code in: src/customers/ If I analyze only src/payments/ , I've removed half of the evidence. Arid 2.0 therefore separates what gets analyzed from what gets reported: arid . --focus src/payments Arid still performs whole-corpus duplicate detection. Baseline enforcement still happens against the complete result, and only afterward does focus filtering determine which groups are reported. If a focused finding also occurs outside the focused path, those occurrences remain part of the finding. In other words, focus changes what you ask Arid to show you without changing the corpus Arid uses to determine whether the code is duplicated. That's an important distinction for CI jobs and coding agents operating on a specific part of a larger repository. Existing Duplicate Debt Is a Lifecycle Baselines were already part of Arid before v2. The idea is straightforward: perhaps you're introducing duplicate-code enforcement into a mature project that already has 300 duplicate groups. You could fix all 300 before adopting the tool. Or don't adopt the tool. Neither is especially compelling. A baseline gives you a third option: accept the existing debt temporarily while preventing new duplicate debt from being introduced. Arid 2.0 extends that into an actual lifecycle. arid . --baseline-status arid-baseline.json can distinguish accepted duplicate debt, active/new findings, and stale baseline entries. Then: arid . --prune-baseline arid-baseline.json removes stale acceptance when the corresponding duplication no longer exists. It never silently accepts new debt. That gives a team a useful progression: existing duplication ↓ baseline it ↓ prevent new duplication ↓ refactor existing duplication over time ↓ prune stale baseline entries ↓ smaller baseline You don't have to make an old codebase perfect before you're allowed to stop making it worse. I suspect that principle applies to considerably more than duplicate code. Failure Doesn't Have to Mean "Tell Me Nothing" Source analysis has another annoying edge case. Imagine scanning 3,000 Python files and one cannot be read, parsed, or normalized. Should the entire analysis disappear? Sometimes yes. If you're enforcing a complete quality gate, an incomplete analysis cannot be treated as success. But that doesn't mean the useful results from the other 2,999 files need to vanish. Arid 2.0 adds: arid . --keep-going --json Independent source failures are collected while valid files continue through detection. The important part is that Arid doesn't pretend partial analysis is complete analysis. A report-v4 result explicitly says: { "complete": false } and includes structured source errors. The process still exits with operational status 2 , and incomplete reports cannot be emitted as SARIF. The intent is simple: produce as much useful information as you safely can, but be explicit about how complete that information is. A human can inspect the partial result, and an automated consumer can make its own decision. Neither has to guess whether the analysis silently skipped something. One Scan, Several Consumers A CI pipeline often wants more than one representation of the same result. Maybe developers want readable console output, the build system wants JSON, GitHub code scanning wants SARIF, and the job summary wants Markdown. The inefficient answer is to run the analyzer four times. Arid 2.0 can instead produce multiple outputs from one in-memory report: arid . \ --format text \ --report json=artifacts/arid.json \ --report markdown=artifacts/arid.md \ --report sarif=artifacts/arid.sarif The source isn't reparsed four times and duplicate detection isn't repeated four times. The analysis happens once, and the result is rendered for the consumers that need it. Obvious in retrospect? Probably. Still worth doing. Machine-Readable Means Having a Contract Once other software starts consuming CLI output, "it happens to be JSON" isn't enough. Arid 2.0 introduces report schema v4 with explicit fields for things like the schema version, tool version, analysis metadata, completion state, structured errors, and finding fingerprints. The schema itself is published: schemas/report-v4.schema.json Arid also publishes contracts for capabilities and fatal JSON-mode operational errors. And: arid --capabilities allows tooling to discover deterministic build capabilities without first discovering or analyzing a project. This is partly about ordinary CI integration, but there's another consumer I care about more now than I would have a few years ago: coding agents. I use AI heavily in my own development workflow. An agent interacting with a tool shouldn't have to scrape human-readable console output and hope a sentence doesn't change in the next release. If we're increasingly going to have software using software on our behalf, the interfaces between those tools need to become more explicit, not less. JSON gives us a machine-readable format. Publishing the schema tells the consumer what that format actually promises. Arid Now Has an Official GitHub Action Of course, the easiest integration is the one you don't have to assemble yourself. Arid 2.0 ships an official composite GitHub Action: - uses: sponge-b0b/arid@v2.0.0 with: paths: . The Action installs the exact Arid release associated with its tag and performs one scan. It can expose core metrics as outputs, write a job summar