dsh-plugin-llm-verifier

A plugin for DeepSeek Harness that adds an LLM verifier: it grades candidate solutions with a model and returns scores between 0 and 1. Based on LLM-as-a-Verifier (paper).

The headline feature is verify_rollout: ask for something once, and the plugin runs several independent agent attempts in parallel, grades them, and gives you the best one.

Install

Requires Node 20+ and a working DeepSeek Harness profile. Install into the profile your dsh command actually loads — web for dsh web, headless for the CLI:

cd ~/.dsh/profiles/web          # or ~/.dsh/profiles/headless, etc.
npm install github:uson1x/dsh-plugin-llm-verifier

Then append this entry to that profile's cordis.patch.yml (the file is a YAML list and usually already exists — add to it, don't replace it; examples/cordis.patch.yml is a copy with the most useful options commented in):

- insert:
    - id: llm-verifier
      name: dsh-plugin-llm-verifier
      config:
        provider: deepseek-official
        model: deepseek-v4-pro

provider and model name the LLM route that does the grading — replace them with a provider and model id your profile registers (the same names dsh's model picker shows). The plugin refuses to load without them.

Restart dsh (patch files are read at boot). If the web app was already open in a browser tab, reload the tab once so it picks up the plugin's UI bundle.

Smoke test: ask the agent to "use llm as a verifier to write a haiku". You should see a verify_rollout call fan out into subagents — and in the web app, a Verifier tab next to Chat and Trajectory.

To update later: re-run the npm install github:… command in the profile and restart dsh.

verify_rollout needs a subagent provider named spawn (present in stock dsh); the other three tools work anywhere.

Use

Just talk to your agent. These all work:

use llm as a verifier to write a landing page tagline

try this 5 times and keep the best: …

here are three drafts — pick the strongest one

The plugin adds a short note to the system prompt so the agent knows to route phrases like these to the right tool. You never have to name a tool.

Each attempt ("rollout") runs as a separate agent session. Open the parent conversation's subagent list (the tree icon in the header) to watch them run and read what each one did.

The four tools

Tool	What it does
verify_rollout(task, n?, rollout_model?)	Run n independent attempts (default 3, allowed 2–8), grade them, return the winner
verify_select(task, candidates[])	You already have N candidates (at least 2); pick the best
verify_compare(task, candidate_a, candidate_b)	Compare exactly two candidates
verify_track(task, trajectory[])	Score how much progress a step-by-step attempt has made

Other plugins can call the same functions directly via ctx.verifier (select, compare, track, score).

Web UI card

…