Computer Use Automation

Computer Use Automation

> Category: Engineering

> Domain: AI Agents

Overview

The Computer Use Automation skill helps you design AI agents that operate a graphical interface the way a person does — take a screenshot, reason about what is on screen, then click, type, scroll, or navigate, and repeat. It covers the core perception→reason→action loop, the decision of when computer-use is the right tool versus a structured API/MCP tool (prefer a real API whenever one exists; reach for computer-use only for GUIs with no programmatic surface), reliability patterns (grounding every action in the *current* screenshot, verifying after each step, recovering from misclicks), safety guardrails (confirmation gates for destructive actions, sandboxing, avoiding blocking dialogs), and how to evaluate a computer-use agent. It is model-agnostic — the patterns apply to any computer-use-capable model and any GUI tool surface.

Clarify First

Before designing or auditing a computer-use agent, confirm these inputs. If any is unknown or vague, ASK — do not assume:

• [ ] Does a real API/MCP tool exist? — whether the target exposes an API, SDK, CLI, or MCP server, or is GUI-only (the single biggest factor; if a real API exists, prefer it and skip computer-use)
• [ ] Task & risk — what the agent must accomplish and whether any step is destructive or irreversible (delete, send, pay, submit), which sets the confirmation gates and sandboxing
• [ ] Which tool — advise on tool choice for a target, or lint a planned action sequence for safety (selects tool_choice_advisor.py vs action_safety_linter.py)

Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.

Quick Start

# Decide computer-use vs API/MCP for a target
python scripts/tool_choice_advisor.py --api-exists no --gui-stability high --volume low --json

# Lint a planned action sequence for safety/reliability gaps
python scripts/action_safety_linter.py --file planned_actions.json

# Read actions from stdin and emit a markdown risk report
echo '[{"type":"click","target":"Delete"},{"type":"submit","target":"Confirm"}]' \
  | python scripts/action_safety_linter.py --format markdown

Tools Overview

| Tool | Purpose | Key Flags |

|------|---------|-----------|

| tool_choice_advisor.py | Recommend computer-use vs structured API/MCP for a target, with rationale | --api-exists, --gui-stability, --volume, --reversible, --json |

| action_safety_linter.py | Scan a planned action list for destructive verbs, missing verification, missing confirmation gates, and dialog-triggering patterns | --file, --format, --json |

All scripts: Python 3 standard library only, argparse CLI, --json and human-readable output. Run --help for full usage.

Workflows

Decide and Design a Computer-Use Agent

• Run tool_choice_advisor.py with the target's API/MCP availability, GUI stability, and volume — if it says "use API/MCP," stop and build against the real interface instead.
• If computer-use is justified, draft the action plan as the screenshot→reason→action loop: each step re-grounds on a fresh screenshot before acting.
• Add a verification observation after every state-changing action (read back the resulting screen, not the intent).
• Insert confirmation gates before any destructive/irreversible step and choose a sandbox (throwaway profile, test account, isolated VM/container).

Audit a Planned Action Sequence

• Express the plan as a JSON/text list of actions (type, target, optional verified/confirmed).
• Run action_safety_linter.py --file plan.json to flag risky verbs, unverified state changes, ungated destructive actions, and dialog-triggering patterns.
• Resolve each finding — add verification steps, add confirmation gates, replace blocking-dialog flows.
• Re-run until clean, then dry-run in the sandbox before any real target.

Reference Documentation

• Computer Use Patterns - The action loop; computer-use vs structured-tool decision matrix; reliability patterns (grounding, verification, recovery); safety guardrails (confirmation gates, sandboxing, blocking dialogs); evaluation approach; and common failure modes.

Common Patterns

Ground Every Action in the Current Screenshot

• Never act on a stale screenshot or a remembered layout — re-capture before each action.
• Reference elements by what is visible now (label, position) rather than a cached coordinate from a prior turn.
• After acting, take a fresh screenshot and confirm the expected change actually happened before continuing.

Gate Destructive Actions and Sandbox by Default

• Require an explicit confirmation step before delete, send, pay, submit, or any irreversible action.
• Run in a sandbox first: throwaway browser profile, test account, or isolated VM/container.
• Avoid flows that spawn blocking modal/native dialogs (file pickers, OS print dialogs) that the agent cannot see or dismiss; prefer paths that keep state on the page.

Prefer the Real Interface When It Exists

• A documented API, SDK, CLI, or MCP tool is more reliable, cheaper, and more verifiable than pixels — use it.
• Reserve computer-use for genuinely GUI-only targets, one-off tasks, or bridging gaps an API does not cover.
• For high-volume or business-critical flows, the cost of computer-use flakiness usually justifies building or requesting an API.