Design AI Confidence as a Recovery Path, Not a Percentage
Start with Consequence
Classify the proposed action before choosing a pattern:
| Consequence | Example | Interface response |
|---|---|---|
| reversible, private | rename a local draft | show preview and undo |
| reversible, shared | open a pull request | show evidence, diff, and reviewer |
| costly to reverse | migrate data | require plan, validation, backup, approval |
| irreversible or external | delete production data | keep deterministic guardrails; never rely on confidence |
A model score cannot authorize a high-consequence action. It can help prioritize review only when calibrated for the same task distribution and clearly explained.
Use an Evidence-and-Recovery Card
Proposed change: Update the cache TTL from 5 to 30 minutes.
Evidence:
- requirement.md, lines 18–22
- config/cache.ts, current value
Uncertainty: The requirement does not specify the staging environment.
Actions: [Edit proposal] [Ask a question] [Approve for staging]
Recovery: Creates a reviewed pull request; does not deploy.
This communicates provenance, a specific gap, bounded authority, and recovery. It remains useful without a percentage.
If a score is necessary, label the event it predicts: “In validation, 81% of suggestions above this threshold required no factual correction.” Link the dataset date, sample size, task definition, threshold, and known failure groups. Never translate model probability directly into “chance this is true.”
Research the Decision, Not Visual Preference
Give participants realistic tasks with low and high consequences. Vary evidence quality and missing information. Observe whether they notice unsupported claims, choose an appropriate action, understand what approval will do, and recover after a deliberately wrong suggestion. Include keyboard and screen-reader use; uncertainty cannot be encoded by color or a tooltip alone.
The public MonkeyCode repository describes AI tasks, project requirements, managed development environments, and team collaboration. Evidence-and-recovery patterns are relevant to that product category, but this is a proposed framework, not a study of MonkeyCode's current interface.
Disclosure: I contribute to the MonkeyCode project. The contextual description is public; no user-research result about MonkeyCode is claimed.
Useful confidence design helps a person inspect, constrain, correct, and recover. If the number does none of those things, remove it.
Comments
No comments yet. Start the discussion.