DEV Community

I put a security scanner inside Jira as an AI agent - the trick was keeping the judgment out of the prompt

Last week I shipped Bastion: a Rovo agent inside Jira that scans any issue for CVE mentions and vulnerable-package references, enriches them against the live NVD API, and produces a 0-100 risk score with recommendations.

It files the remediation ticket without leaving the workflow where the work already lives. The interesting engineering decision wasn't the LLM part. It was what I deliberately didn't let the model decide.

The split: model narrates, code decides

Bastion is an agent, but the security judgment is a pure function:

score += cveData.filter(c => c.severity === 'CRITICAL').length * 40;
score += cveData.filter(c => c.severity === 'HIGH').length * 20;
score += Math.min(cves.length * 5, 20);
return Math.min(score, 100);

CRITICAL findings are worth 40 points, HIGH 20, each additional unique CVE adds 5 (capped at 20), hard ceiling at 100. Severity bands: 70+ CRITICAL, 40+ HIGH, 15+ MEDIUM.

The model reads the issue text, the scanner extracts CVE IDs with a regex, NVD returns the CVSS data, and this arithmetic produces the number. The LLM writes the explanation - "this looks like Log4Shell exposure, patch this sprint" - but it cannot inflate or deflate the score.

Judgment probabilistic, boundary deterministic: same lesson as prompt-injection defense, applied to risk scoring.

The unglamorous parts that actually mattered

Rate limits are the real API. NVD's public API returns 429/403 under load. The client retries once after a 6-second backoff, caches each CVE for 6 hours, and spaces lookups 1.2 seconds apart. Without that, the "live lookup" demo dies on the third question.

Extraction is regex, and that's honest. CVE IDs are CVE-\d{4}-\d{4,7} - trivially extractable. Package names are a fixed list (log4j, openssl, struts, the usual suspects). The honest limitation: a vulnerable dependency not on the list gets missed. I documented it in the README rather than pretending coverage we don't have. An agent that silently under-reports is worse than one that admits its blind spots.

Forge constraints shape the design. No direct DB, no long-running processes - so state lives in Jira itself (labels, issue fields) and the daily sweep is a scheduled trigger, not a daemon. If your platform has constraints, let them pick the architecture instead of fighting them.

What I'd do differently

Package detection should come from the project's dependency manifest (package.json, requirements.txt, pom.xml) when the issue links a repo - regex on prose will always have false negatives.

And the scoring weights (40/20/5) are defensible but arbitrary; they should be tuned against real triage outcomes, which is exactly what a vuln-triage scoring worksheet is for.

Where this connects

  • Bastion is open source: github.com/wesleygitari2-cloud/bastion - manifest, resolver, security-core, and the React panel are all there.
  • Free: the Security Triage Checklist covers the manual version of this workflow (KEV/EPSS checks, scoring, SLAs) - useful even if you never ship an agent.
  • Paid: the Vuln Triage System ($14, now also in Spanish at $12) is the scoring model + remediation policy + reporting templates Bastion's arithmetic is a simplified version of. Store link auto-updates; pay in BTC, no account, instant download.
Read on DEV Community ↗ ← Back to News

Comments

No comments yet. Start the discussion.