Confidence Is Not Authorization: A Safer GitHub Issue-Automation Policy

GitHub's new issue automations can now label, type, assign, or close issues with a confidence level and a rationale. That sounds like a small UX improvement. It is actually a policy-design problem: a model's confidence is a guess about its own output, while authorization is a rule about what the system is allowed to change. If you let those concepts collapse into one threshold, you can build an automation that is easy to operate and hard to govern. This guide turns the preview into a reviewable policy: automate low-impact metadata only when the action is reversible, keep ambiguous changes in a suggestion queue, and put security-sensitive decisions behind a real control boundary.

Scope note: GitHub describes rationale, confidence, and approvals as a public preview. The examples below are a policy template, not a claim that the preview is a security product.

What GitHub actually added

GitHub's July 23 changelog describes three controls for supported issue actions:

• Approvals: ask the automation to suggest a change instead of applying it immediately.
• Confidence: the agent labels an action high, medium, or low confidence. High-confidence changes can apply automatically; medium and low-confidence changes wait as suggestions.
• Rationale: each action records why it was proposed or applied.

The initial action set covers labels, fields, issue type, closing, and assignees. You can search for waiting items with has:suggestions, and repository administrators can configure the automation level that sets the confidence threshold.

The important limitation is in GitHub's own wording: approvals are a workflow convenience, not a security control. If an agent has permission to change issues, it can apply changes directly instead of suggesting them. A review panel improves the workflow, but it does not replace permissions, branch protection, environment gates, or server-side authorization.

The policy mistake to avoid

Here is the tempting rule:

if confidence == "high": apply_change()
else: ask_a_human()

That rule treats confidence as if it were permission. It is not. A high-confidence label can still be the wrong label, and a correctly classified issue can still be unsafe to close or assign. The risk is not only model error. It is also stale repository policy, ambiguous ownership, duplicate reports, and a permission that is broader than the workflow author intended.

A safer decision has at least four inputs:

• Action impact: what happens if the change is wrong?
• Evidence quality: what facts support the action, and are they still current?
• Reversibility: can a maintainer undo it without losing information or hiding work?
• Authority: is this actor and this workflow actually allowed to perform the action?

Confidence can help rank work inside that decision. It should not make the authority decision for you.

A practical triage matrix

Use a policy like this as a starting point, then tune it against false positives from your own issue tracker:

The matrix deliberately treats "close" and "assign" differently from a harmless topic label. The model may be very sure about its classification and still lack the authority or context to make the consequential change.

Configure the workflow around intent, not just output

For GitHub Agentic Workflows, the changelog documents issue-intents: true for supported safe outputs such as set-issue-type, set-issue-field, add-labels, close-issue, assign-to-agent, and assign-to-user.

A minimal policy sketch could look like this:

---
name: Triage incoming issues
on:
  issues:
    types: [opened, edited]
permissions:
  issues: write
issue-intents: true
---

# Keep the model's proposed action narrow.
# Require a rationale that names the evidence it used.
# Prefer a suggestion for assignment, closure, or escalation.

This is not a complete security boundary. The issues: write permission is still consequential, and the policy comments are not enforcement. Treat the workflow as an orchestrator whose permissions and repository settings need independent review.

Make suggestions auditable

For every applied or suggested action, retain a small record with:

• issue number and repository;
• action, old value, and proposed value;
• confidence level;
• rationale and the evidence fields used;
• workflow version and model/tool version when available;
• actor permissions at execution time;
• reviewer decision, timestamp, and any correction.

Rationale is useful only if it can be compared with the action. "This looks like a bug" is not enough for an assignment or closure. A stronger record names the signal, for example: "The issue mentions payments/checkout, the component map maps that path to Payments, and the current owner is team-payments." Even then, the record is evidence for review, not proof that the change was authorized.

Test the policy with a replay set

Before changing the confidence threshold, create a small labeled set from real issues. Keep the set representative rather than enormous:

issue_id,expected_action,expected_value,review_required
101,add-label,bug,false
102,assign-team,payments,true
103,no-change,,true
104,close-duplicate,101,true
105,add-label,security,true

Replay it after changing the prompt, workflow, model, label taxonomy, or ownership map. Track separate outcomes:

• action precision: when the automation acted, how often was the action correct?
• action recall: how often did it find actions that should have happened?
• abstention quality: when it suggested or skipped, did it avoid a costly mistake?
• review load: how many suggestions were actionable versus noise?
• reversal rate: how often did maintainers undo an applied change?

Do not optimize only for the percentage of issues processed automatically. A lower automation rate can be the better result if it removes dangerous false positives and keeps review focused on high-impact work.

Failure modes worth adding to the test set

Include cases that exercise the policy, not only the happy path:

• a stale component-to-owner mapping;
• a duplicate issue whose "canonical" issue was already closed;
• a prompt injection in an issue body or pasted log;
• a high-confidence classification with a missing required field;
• a low-confidence suggestion that would be harmless to apply;
• a security label inferred from an untrusted user claim;
• an issue that changes meaning after an edit;
• a workflow with write permission that should have been read-only.

For high-impact actions, test the permission boundary separately. An approval panel is not a substitute for verifying that a workflow cannot perform an action outside its intended scope.

A short rollout plan

• Start in suggestion mode. Use has:suggestions to build a review queue and inspect rationales.
• Measure reversals and review cost. Do not raise the threshold because the queue feels busy; inspect which suggestions are actually safe.
• Automate only reversible metadata first. Keep closure, assignment, escalation, and security labels in review until the evidence and authority rules are stable.
• Separate policy from prompt text. Store ownership, allowed labels, and action boundaries in versioned configuration or code.
• Re-test after every taxonomy or permission change. A label rename or team change can invalidate an apparently good automation.

Checklist

• [ ] Confidence is treated as a ranking signal, not authorization.
• [ ] Each action has an explicit impact and reversibility class.
• [ ] The workflow permission is narrower than the broadest possible agent capability.
• [ ] Suggestions and applied actions retain rationale and evidence.
• [ ] Closure and assignment have a review path and a rollback path.
• [ ] Prompt-injection, stale-data, and missing-context cases are in the replay set.
• [ ] Threshold changes are evaluated against precision, recall, abstention, review load, and reversals.
• [ ] A human reviews the policy itself, not only individual suggestions.

GitHub's preview makes issue automation easier to inspect, which is valuable. The safe interpretation is not "the agent is confident, so let it act." It is "the agent produced a proposal with evidence; now apply the repository's policy for impact, authority, and review."

Sources

• GitHub: Agent automation controls in GitHub Issues in public preview
• GitHub Docs: About rationale, confidence, and approvals for issues
• GitHub Docs: About GitHub Agentic Workflows

How are you separating model confidence from actual authorization in your issue or repository automations?