Computer-use agents interact with software the way people do: they take screenshots, decide where to click or what to type, and repeat.
How They Work
The harness captures the screen, sends it to a multimodal model, and executes the model's chosen action — click at coordinates, type text, press keys, scroll. The loop continues until the task is done.
When They Help
- Automating applications without APIs.
- Testing user interfaces.
- Tasks spanning several applications.
Limitations
- Slower and more expensive than API-based tools.
- Errors in reading screens or clicking the wrong element.
- Fragile with pop-ups, changing layouts and unexpected dialogues.
Prefer APIs and structured tools where available; use screen control when there's no alternative.
Security Risks
Screens show untrusted content — web pages, emails, documents — that may contain instructions aimed at the agent. Computer-use agents may also see sensitive information.
Precautions
- Run in an isolated virtual machine or container.
- Limit logged-in accounts and available credentials.
- Restrict network access.
- Require confirmation for purchases, sending messages and account changes.
- Monitor sessions and keep them short.