Synod Update: Adding a Deterministic Safety Net (and Proving It Helps)

What Changed

Quick update on Synod, the multi-agent code reviewer I've been building for the Qwen Cloud hackathon. I added a Semgrep pre-filter in front of the security agent.

Before, every finding came purely from the LLM reading the code and reasoning about it - which works, but LLMs are stochastic. Same file, different run, sometimes a different result.

Now Semgrep scans first with deterministic rules, and the security agent validates and enriches those candidates instead of starting from zero every time.

Did It Actually Help?

I was skeptical of my own change, so I benchmarked it properly instead of assuming. Ran a single-agent baseline against the full council, with and without the pre-filter, same vulnerable file, checked against known ground truth:

The interesting part wasn't the top-line numbers - it was that the single-agent and LLM-only council both had real run-to-run variance. Sometimes it caught everything, sometimes it missed half. That's not a reviewer you can trust in CI.

Adding the deterministic scanner as a floor fixed that. It's not smarter, it's just consistent - and consistency turned out to matter more than I expected.

Also Shipped

• A GitHub webhook - open a PR, Synod reviews the diff and comments directly, findings grouped by severity.
• A small CLI, closer to how tools like Claude Code feel in the terminal, for reviewing files or whole directories without touching the API directly.

Repo's still open source: github.com/02NIN20/Synod

Built for the Global AI Hackathon Series with Qwen Cloud - Track 3: Agent Society.