红尘
d73ff9b0fb
Deploy Site / deploy-vercel (push) Has been cancelled
Deploy Site / deploy-docs (push) Has been cancelled
Docker / shell lint / Lint Dockerfile (hadolint) (push) Has been cancelled
Docker / shell lint / Lint docker/ shell scripts (shellcheck) (push) Has been cancelled
Docker Build and Publish / build-amd64 (push) Has been cancelled
Docker Build and Publish / build-arm64 (push) Has been cancelled
Lint (ruff + ty) / ruff + ty diff (push) Has been cancelled
Lint (ruff + ty) / ruff enforcement (blocking) (push) Has been cancelled
Lint (ruff + ty) / Windows footguns (blocking) (push) Has been cancelled
Nix Lockfile Fix / auto-fix-main (push) Has been cancelled
Nix Lockfile Fix / fix (push) Has been cancelled
Nix / nix (macos-latest) (push) Has been cancelled
Nix / nix (ubuntu-latest) (push) Has been cancelled
OSV-Scanner / Scan lockfiles (push) Has been cancelled
Build Skills Index / build-index (push) Has been cancelled
Tests / test (1) (push) Has been cancelled
Tests / test (2) (push) Has been cancelled
Tests / test (3) (push) Has been cancelled
Tests / test (4) (push) Has been cancelled
Tests / test (5) (push) Has been cancelled
Tests / test (6) (push) Has been cancelled
Tests / e2e (push) Has been cancelled
uv.lock check / uv lock --check (push) Has been cancelled
Docker Build and Publish / merge (push) Has been cancelled
Build Skills Index / trigger-deploy (push) Has been cancelled
Tests / save-durations (push) Has been cancelled
44 lines
1.9 KiB
Python
44 lines
1.9 KiB
Python
"""Computer use toolset — universal (any-model) macOS desktop control.
|
|
|
|
Architecture
|
|
------------
|
|
This toolset drives macOS apps through cua-driver's background computer-use
|
|
primitive (SkyLight private SPIs for focus-without-raise + pid-scoped event
|
|
posting). Unlike #4562's pyautogui backend, it does NOT steal the user's
|
|
cursor, keyboard focus, or Space — the agent and the user can co-work on the
|
|
same machine.
|
|
|
|
Unlike #4562's Anthropic-native `computer_20251124` tool, the schema here is
|
|
a plain OpenAI function-calling schema that every tool-capable model can
|
|
drive. Vision models get SOM (set-of-mark) captures — a screenshot with
|
|
numbered overlays on every interactable element plus the AX tree — so they
|
|
click by element index instead of pixel coordinates. Non-vision models can
|
|
drive via the AX tree alone.
|
|
|
|
Wiring
|
|
------
|
|
* `tool.py` — registers the `computer_use` tool via tools.registry.
|
|
* `backend.py` — abstract `ComputerUseBackend`; swappable implementation.
|
|
* `cua_backend.py`— default backend; speaks MCP over stdio to `cua-driver`.
|
|
* `schema.py` — shared schema + docstring for the generic `computer_use`
|
|
tool. Model-agnostic.
|
|
* `capture.py` — screenshot post-processing (PNG coercion, sizing, SOM
|
|
overlay if the backend did not).
|
|
|
|
The outer integration points (multimodal tool-result plumbing, screenshot
|
|
eviction in the Anthropic adapter, image-aware token estimation, the
|
|
COMPUTER_USE_GUIDANCE prompt block, approval hook, and the skill) live
|
|
alongside this package. See agent/anthropic_adapter.py and
|
|
agent/prompt_builder.py for the salvaged hunks from PR #4562.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
# Re-export the public surface so `from tools.computer_use import ...` works.
|
|
from tools.computer_use.tool import ( # noqa: F401
|
|
handle_computer_use,
|
|
set_approval_callback,
|
|
check_computer_use_requirements,
|
|
get_computer_use_schema,
|
|
)
|