Hermes Computer Use: How to Set Up a Background AI Agent That Works While You Keep Typing (2026)
Verdict: Hermes Agent's Computer Use feature is the most practical background desktop automation tool available in 2026. It lets an AI agent click, type, scroll, and drag across your real applications without moving your cursor, stealing keyboard focus, or switching virtual desktops. You and the agent work on the same machine at the same time. It is free, open source (MIT), works on macOS, Windows, and Linux, and runs with any visionâcapable model - Claude, GPT, Gemini, or a local open model. Setup takes one command and about five minutes. Last verified: 2026â08â04 ¡ Free and open source (MIT) ¡ Works on macOS, Windows, Linux ¡ Any visionâcapable model ¡ ~5âŻmin setup
Volatile facts: model pricing, cuaâdriver version, and platform support change often - last checked August 2026.
What Is Hermes Computer Use?
Hermes Computer Use is a feature of Hermes Agent - an openâsource personal AI agent built by Nous Research - that gives the agent the ability to drive your real desktop. It can click buttons, type text, scroll pages, drag elements, and read what is on your screen, all through the operating system's native accessibility APIs. The key differentiator is that everything happens in the background. Your real mouse cursor does not move. Your active window does not jump to the front. Your virtual desktop or macOS Space does not switch. You can keep typing in your editor while the agent clicks around in a browser in another window. This is fundamentally different from older automation tools like pyautogui or AutoHotkey, which take over your mouse and keyboard.
Under the hood, Hermes speaks the Model Context Protocol (MCP) over standard I/O to cuaâdriver, an openâsource background computerâuse driver licensed under MIT. The driver uses each platform's native accessibility and input stack:
- macOS - AX (private SkyLight SPIs)
SLPSPostEventRecordTo- pidâscoped, no cursor warp. Events posted directly to target processes. - Windows - UIAutomation
SendInput+PostMessage- no focus steal. Works without elevation for nonâUAC windows. - Linux - ATâSPI (X11 + Wayland)
XTest(X11) / virtualâkeyboard (Wayland). XWayland bridge needed on pure Wayland.
The result on every platform is the same: the agent reads the accessibility tree of any visible window and posts synthesized input events without bringing windows to the front or moving the real cursor (Hermes Computer Use docs, cua.ai noâforeground contract).
How Does Background Computer Use Work?
It works by routing synthetic input events to specific process IDs instead of the shared system cursor.
- macOS -
cuaâdriveruses private SkyLight SPIs (SLPSPostEventRecordTo) to post events directly to the target application - there is no HID event tap and no cursor warp. - Windows - It uses
SendInputcombined withPostMessageto dispatch input without focus theft. - Linux - It uses the
XTestextension for X11 or a virtualâkeyboard interface for Wayland compositors.
Each action the agent takes - a click, a type, a scroll - is scoped to a specific window and process. The agent captures a screenshot (with a setâofâmarks overlay showing every clickable element numbered), decides what to click, sends the event, then captures again to verify. A faint tinted overlay cursor appears on screen to show where the agent is acting, but your real OS cursor remains wherever you left it (Hermes docs, trycua/cua on GitHub).
A 20âaction session on a 1568Ă900 display typically costs about 30âŻK tokens of screenshot context, not 600âŻK, because Hermes prunes old screenshots from context and estimates each image at ~1âŻ500 tokens (Hermes Computer Use docs).
How to Set Up Hermes Computer Use (Step by Step)
Setting it up takes about five minutes. The prerequisite is that Hermes Agent itself is already installed - if you do not have it yet, run the oneâline installer.
Install the computerâuse driver
Open your terminal and run:hermes computer-use installThis fetches and runs the upstream cuaâdriver installer -
install.shon macOS and Linux,install.ps1on Windows. You can verify the install with:hermes computer-use statusAlternatively, you can enable it interactively: run
hermes tools, pick âComputer Use (macOS/Windows/Linux)â, and the setup runs the same upstream installer (Hermes docs).Grant platform permissions
After install, grant the OSâlevel permissions the driver needs:- macOS - System Settings â Privacy & Security â Accessibility + Screen Recording â allow your terminal (or the Hermes desktop app). Run
hermes computer-use doctorto see which permission is missing. - Windows - No permissions needed at install time. If driving over SSH (not RDP or console), you need the autostart pattern for the SessionâŻ0 â SessionâŻ1+ proxy (cua.ai/docs).
- Linux - Ensure
DISPLAYis set for X11 sessions, orXDG_SESSION_TYPE=waylandfor Wayland. Pure Wayland needs an XWayland bridge for screen capture. ATâSPI must be enabled (default on GNOME, KDE, Xfce).
- macOS - System Settings â Privacy & Security â Accessibility + Screen Recording â allow your terminal (or the Hermes desktop app). Run
Start a session with computer use enabled
hermes -t computer_use chatOr add
computer_useto your enabled toolsets in~/.hermes/config.yamlso it is always available.Verify with a simple prompt
Give the agent a simple instruction like âopen my notes app and type âhello worldââ - or, for a safer test, âtake a screenshot of my desktop and tell me what windows are open.â You should see the tinted agent cursor move while your real cursor stays put.Run diagnostics if anything fails
hermes computer-use doctorThis runs cuaâdriver's structured health check and prints a perâcheck matrix. It tells you exactly what is wrong - a missing permission, a stale binary, a display server problem. It is the first thing to run when anything is off (Hermes docs).
Which Models Work With Hermes Computer Use?
Hermes Computer Use works with any toolâcapable model that can process screenshots. It does not lock you into Anthropic's native computerâuse schema the way some integrations do. This is one of its biggest advantages.
| Provider | Vision? | Works? | Notes |
|---|---|---|---|
| Anthropic (Claude Sonnet/Opus 3+) | Yes | Yes | Best overall; setâofâmarks + raw coordinates |
| OpenAI (GPTâ4+, GPTâ5) | Yes | Yes | Multiâpart tool messages supported |
| Google (Gemini 2+) | Yes | Yes | Toolâcalling and vision both supported |
| OpenRouter (any vision model) | Yes | Yes | Route to whichever model fits the task |
| Local vLLM / LM Studio / Ollama | Yes | Yes | If the model supports multiâpart tool content |
| Textâonly models | No | Degraded | Use mode="ax" for accessibilityâtreeâonly operation |
Source: Hermes Computer Use provider compatibility
The practical implication: you can run routine clicking and typing tasks through a free or cheap model and save premium models for reasoningâheavy steps. This keeps your total cost near zero if you use a freeâtier model from OpenRouter or a local model on your own hardware.
What Safety Guardrails Does Hermes Computer Use Have?
Hermes Computer Use has builtâin guardrails designed to keep you in control. The system operates in two permission modes:
- Manual/smart approvals (default) - Any potentially destructive action - deleting a file, running a shell command - stops and asks for your approval before executing. You see what the agent wants to do and can approve, deny, or stop midâtask. Best for daily work; everything except throwaway VMs.
- YOLO / unrestricted - One explicit risk acceptance; no runtime approval prompts. The unrestricted daemon is private to that session and ends when the session closes. Disposable VMs only; never on your real machine.
In the default mode, Hermes also hardâblocks dangerous patterns: it will not type passwords, it will not execute shell payloads matching dangerous pattern lists, and it blocks actions like emptying the trash or logging you out. Promptâinjection protections are layered in - the agent is instructed not to follow sneaky instructions hidden inside screenshots or web pages (Hermes docs). YOLO mode explicitly does not protect against prompt injection or unintended input. The official documentation recommends using it only inside a disposable VM with accounts and data whose full compromise you accept.
What Can You Actually Do With It?
The practical use cases for background computer use fall into a few categories - all of which can run while you keep working on something else on the same machine:
- Email triage and drafting - âRead my inbox, identify the three emails that need replies today, draft the replies, and save them to my Drafts folder.â
- Noteâtaking and journaling - âOpen Notes, capture a journal entry about today's decisions, and tag it with the right project.â
- File organization - âMove every PDF on my Desktop into the correct project folder based on filename context, and rename them consistently.â
- Knowledgeâmanagement maintenance - If you use a tool like Obsidian as a memory layer, point Hermes at your vault to organize and link notes while you focus on content. For more on this pattern, see our guide to building an AI agent OS with a threeâlayer memory blueprint.
- Crossâapp workflows - âTake this morning's meeting notes, summarize the action items, paste them into the project tracker, and create followâup tasks.â This is exactly the kind of multiâstep, crossâapp shuffle that used to eat 30 minutes of your day.
- Remote work - Firing off tasks from your phone while your agent keeps working on your machine back home - Hermes connects to messaging platforms like Telegram and Discord. You can approve actions from your phone and let the agent carry on. See our guide on accessing your AI agent dashboard remotely with Tailscale for the networking layer.
Can You Run Multiple Agents at Once?
Yes. Each Hermes run gets its own session ID and its own agent cursor on screen. You can have multiple agents working concurrently without them tripping over each other - one sorting files in the background while another writes notes. Hermes is also smart about screenshot management: it prunes old screenshots from context, so even long multiâaction sessions do not slow your machine to a crawl. This multiâsession capability is part of what makes Hermes a genuine agent platform, not a singleâshot toy.
If you want to extend this further into a full multiâagent orchestration setup, see our guide to running an AI agent operating system and our walkthrough of building a free AI agent team with Hermes Agent and OpenCode.
How Does Hermes Computer Use Compare to Other Tools?
| Tool | Background mode | Free / open source | Crossâplatform | Any model | Permission guardrails |
|---|---|---|---|---|---|
| Hermes Computer Use | Yes | Yes (MIT) | macOS, Windows, Linux | Yes | Yes - destructive actions require approval |
| Anthropic Claude Computer Use | No (foreground) | Paid API | macOS only | Claude only | Limited |
| OpenAI Operator | No (foreground, web only) | Paid | Web only | GPT only | Yes |
| Manus | No | Paid | Web only | Proprietary | Yes |
| pyautogui / AutoHotkey | No - steals cursor | Free / open source | Crossâplatform | No AI model | No |
Hermes is the only tool in the table that combines all four: runs quietly in the background, costs nothing, works crossâplatform, and is not locked to a single model provider.
Sources: Hermes docs, trycua/cua on GitHub, Nous Research.
What Are the Limitations?
- Background mode is slower than foreground. Accessibilityârouted events take ~5-20âŻms on macOS, ~3-10âŻms on Windows, ~5-15âŻms on Linux versus direct HID posting. Not noticeable for agentâspeed clicking, but not suitable for realâtime speedâruns (Hermes docs).
- Complex multiâstep tasks are less reliable. Scope tasks to 3-5 steps for clean output. Very long unattended chains can hit stale element references or unexpected dialogs.
- macOS private SPI risk. Apple can change SkyLight's symbol surface in any OS update. Hermes warns when the installed cuaâdriver is older than its tested baseline, but if you need reproducibility across a macOS bump, pin the driver version with the
HERMES_CUA_DRIVER_VERSIONenvironment variable. - Windows SSH sessions. SSH runs in SessionâŻ0, which has no interactive desktop. Drive from an RDP or console session, or set up the autostart pattern.
- Linux Wayland. Pure Wayland without XWayland will produce empty captures. Use an X11 session or ensure the XWayland bridge is active.
- Token cost on premium models. Screenshots are not free. A 20âaction session is ~30âŻK tokens, which adds up on Claude Opus or GPTâ5. Route routine work to free or local models and save premium tokens for reasoningâheavy steps.
What This Means for You
If you spend a meaningful part of your day on repetitive desktop tasks - sorting files, drafting routine emails, moving data between apps, organizing notes - Hermes Computer Use gives you a free second pair of hands that never gets bored and never needs a break. The fiveâminute setup and anyâmodel flexibility mean there is no barrier to trying it today. Start with a free model for routine work, give the agent goals (not just tiny tasks), and approve its actions while you keep doing the work that actually requires your brain.
For a deeper journey into building a full agent operating system around Hermes - with persistent memory, scheduled automation, and multiâplatform access - see our guide to building a selfâimproving AI agent operating system and our walkthrough of the agent OS that ships any AI tool on demand.
FAQ
Q: Is Hermes Computer Use free?
A: Yes. Hermes Agent and cuaâdriver are both MITâlicensed open source. You only pay for model API calls, and you can use free models from OpenRouter or run a local model on your own hardware at zero cost.
Q: Does Hermes Computer Use work on Windows and Linux?
A: Yes. As of 2026, it works on macOS, Windows, and Linux. Windows needs no special permissions for console sessions; Linux requires a reachable display server (X11 or Wayland with XWayland).
Source: Hermes Computer Use docs.
Q: How long does setup take?
A: About five minutes: one command to install the driver, then grant OSâlevel permissions (Accessibility + Screen Recording on macOS). Run he
Comments
No comments yet. Start the discussion.