dsh-computer-use

<h1 align="center">DSH Computer Use</h1>

Text-first browser and background macOS control for DSH.<br> Target the right process and window without taking over the user's pointer.

<a href="README.zh.md">简体中文</a> · <a href="documentation/architecture.md">Architecture</a> · <a href="documentation/distribution.md">Distribution</a> · <a href="SECURITY.md">Security</a>

Install

Homebrew

brew tap zrui-c/tap
brew trust zrui-c/tap
brew install --cask dsh-computer-use
open -a "DSH Computer Use"

DMG

1. Download the latest DSH-Computer-Use-*-universal.dmg from Releases.
2. Drag DSH Computer Use into Applications and open it.

With either method, authorize Accessibility and Screen Recording, select Install under DSH Plugin, then restart the running DSH Host.

The Homebrew Cask and official DMG install the same Universal 2, Developer ID signed, Apple-notarized app. Users do not need Xcode, Swift, or this source checkout. DSH and Google Chrome must already be installed.

What it does

Surface	Perception	Input
Chromium	Playwright, CDP accessibility/DOM, frames, tabs, optional OCR	Ref-pinned navigation, pointer, keyboard, forms, scroll, drag, upload
macOS	Accessibility tree first, Vision OCR for semantic gaps, independent window capture	AX actions, targeted SkyLight/CoreGraphics, global HID only without a target

Every action returns a fresh bounded text observation. The model works with roles, names, values, state, geometry, and stable snapshot refs instead of assuming it can inspect screenshots. UI and OCR strings are explicitly untrusted data.

Background macOS control

• Carries PID, WindowServer window ID, AX window frame, and element identity through every action.
• Prefers semantic AX actions before coordinate input.
• Routes supported pointer and keyboard events directly to the target process/window.
• Uses a click-through software cursor; targeted actions do not move the physical pointer.
• Keeps post-action observation pinned to the previous target, even while another app stays active.
• Falls back to public CoreGraphics or fails closed when a private capability is unavailable.

Platform boundary

ScreenCaptureKit is public API. The optional background input path dynamically loads private SkyLight symbols and is intended for Developer ID distribution, not the Mac App Store. Private APIs are unsupported by Apple and can change between macOS releases.

On macOS 26, Stage Manager may expose a shelved window only as a small WindowServer thumbnail. Constructing a capture filter for that representation can abort inside SkyLight. DSH Computer Use detects the geometry mismatch first, preserves AX observation, returns an explicit warning, and never stretches a thumbnail into a fake full-window screenshot.

DSH integration

The embedded package declares a DSH bundle in package.json. The setup center runs the official equivalent of:

dsh plugin --profile web add --save-exact file:/path/to/DSH\ Computer\ Use.app/Contents/Resources/Plugin

…