dsh-plugin-midscene

English | 中文

AI-driven UI automation for DeepSeek Harness (DSH), powered by Midscene. The model sees the screen, locates elements by natural-language description, and acts on real targets — a real Android device or a real Chrome browser.

One capability seam (ctx.midscene), two providers, two tools:

	Provider entry	Tool	Target
Android	dsh-plugin-midscene/android	android_ui	one ADB-connected device
Web	dsh-plugin-midscene/web	web_ui	the active page of an already-running Chrome

Each tool is a single tool with an action parameter (like str_replace_editor): the model picks an action (tap / act / input / query / assert / boolean / back) and the tool dispatches internally — no tool-surface bloat.

Requirements

• DSH (dsh CLI) with a profile
• Android: adb devices shows the device
• Web: Chrome started with --remote-debugging-port=9222 --user-data-dir=<dir>; the provider connects, never launches
• A Midscene-compatible vision model, configured through environment variables (see Model configuration)

Install

dsh plugin --profile mysetup add dsh-plugin-midscene

The bundle's default layer registers the two tools. They read ctx.midscene opportunistically, so they appear even before a provider is configured — calling one without a provider fails with a message naming the missing row.

Then add exactly one provider row to your profile's cordis.patch.yml (~/.dsh/profiles/mysetup/cordis.patch.yml; both providers cannot own ctx.midscene in the same context):

Android

- insert:
    - id: midscene-android
      name: dsh-plugin-midscene/android
      config:
        deviceId: ''            # empty: first device from getConnectedDevices()
        aiActionContext: ''     # free-form context for aiAct planning, e.g. app conventions

Web

- insert:
    - id: midscene-web
      name: dsh-plugin-midscene/web
      config:
        browserWSEndpoint: 'ws://127.0.0.1:9222/devtools/browser/<id>'
        aiActionContext: ''

Ready-to-paste provider rows live in examples/.

Get the endpoint from http://127.0.0.1:9222/json/version → webSocketDebuggerUrl. Note the id changes every time that Chrome restarts — update the row and restart dsh.

On teardown the provider destroys its agent and then disconnect()s — never close()s — the browser: the Chrome process belongs to your deployment and keeps running.

Start:

dsh --profile mysetup          # add --port 3081 if 3080 is occupied

Install troubleshooting

dsh plugin add fails with ERR_PNPM_IGNORED_BUILDS naming sharp / @ffmpeg-installer/linux-x64: pnpm ≥ 10 blocks those transitive install scripts (pulled in by @midscene/*) until they are declared. Fix: open ~/.dsh/profiles/<name>/pnpm-workspace.yaml, set the keys pnpm printed under allowBuilds to false (the plugin works without them — flip to true only if you want the sharp/ffmpeg binaries for real-device capture), then re-run the add command. This is a one-time fix per profile.

Tool reference

android_ui and web_ui share one shape:

…