dsh-video-lens

Video understanding for DeepSeek Harness — give text-only agents eyes and ears on video.

A DeepSeek Harness (DSH) plugin that lets text-only LLM agents understand local video files. It provides two tools:

Tool	What it does
video_probe	Cheap, instant metadata via ffprobe: container, duration, resolution, fps, codecs, audio tracks, subtitles.
video_analyze	Content understanding: scene-change-aware frame sampling (ffmpeg scdet), optional ASR transcript (speech with timestamps), fused with any OpenAI-compatible vision model into structured evidence JSON.
video_ask	Time-anchored Q&A: parses explicit time references ("at 3:20", "第2分钟") or locates relevant speech via transcript keyword matching, re-samples frames from the matched windows, and answers with grounded evidence (answer + confidence + supporting timestamps).

v0.3.2. The plugin never locks you into a provider: vision and ASR are both OpenAI-compatible endpoints configured via baseUrl + model + key env var.

How it works

video file ──► video_probe ──► ffprobe ──► compact metadata JSON
           └─► video_analyze ──► scdet scene detection ──► shot boundaries
                                 ├─► ffmpeg frame sampling (one representative frame per shot, capped)
                                 ├─► ffmpeg audio extract ──► ASR transcript (timestamped)   [optional]
                                 └─► OpenAI-compatible vision API ──► evidence JSON

• Scene changes are detected with ffmpeg's scdet filter (ffmpeg ≥ 6.0). Videos without detectable cuts fall back to uniform midpoint sampling.
• ASR is strictly additive: if asrApiKeyEnv is unset or the provider fails, the visual analysis still completes and transcript is null.
• All media work is delegated to ffmpeg/ffprobe on PATH — no native decoding in the agent.

Install

Prerequisites: Node.js ≥ 20, ffmpeg ≥ 6.0 (recommended) with ffprobe on PATH (brew install ffmpeg / apt install ffmpeg).

Option A — npm (recommended)

# in your DSH profile directory (the one containing package.json)
pnpm add dsh-video-lens

Option B — from source (development)

Clone the repo, then mount it into your DSH profile via a local link:

git clone https://github.com/dundunhan/dsh-video-lens.git

Either way, register the bundle in your profile's package.json — this exact block is the full profile configuration:

{
  "dependencies": {
    "dsh-video-lens": "^0.3"
  },
  "dsh": {
    "profile": {
      "bundles": [
        "@deepseek-ai/dsh-base",
        "@deepseek-ai/dsh-web-app",
        "dsh-video-lens"
      ]
    }
  }
}

Then export the keys and restart the profile:

export VIDEO_LENS_API_KEY=sk-...        # vision
export VIDEO_LENS_ASR_KEY=sk-...        # optional, ASR

Do not install the host runtime yourself. @deepseek-ai/dsh-tools is declared as an optional peer: the plugin always uses the dsh-tools that already ships with your DSH installation / DSH Desktop. Adding it to your profile as a dependency — or pinning one exact -rc version, which is what 0.3.1 did — installs a second, older runtime next to the host's, makes the Loader entry fail to import, and takes the whole plugin tree (and the app) down with it.

Configuration

All options are DSH config values:

…