dsh-video-lens

Video understanding for DeepSeek Harness — give text-only agents eyes and ears on video.

A DeepSeek Harness (DSH) plugin that lets text-only LLM agents understand local video files. It provides two tools:

Tool	What it does
video_probe	Cheap, instant metadata via ffprobe: container, duration, resolution, fps, codecs, audio tracks, subtitles.
video_analyze	Content understanding: scene-change-aware frame sampling (ffmpeg scdet), optional ASR transcript (speech with timestamps), fused with any OpenAI-compatible vision model into structured evidence JSON.
video_ask	Time-anchored Q&A: parses explicit time references ("at 3:20", "第2分钟") or locates relevant speech via transcript keyword matching, re-samples frames from the matched windows, and answers with grounded evidence (answer + confidence + supporting timestamps).

v0.3.1. The plugin never locks you into a provider: vision and ASR are both OpenAI-compatible endpoints configured via baseUrl + model + key env var.

How it works

video file ──► video_probe ──► ffprobe ──► compact metadata JSON
           └─► video_analyze ──► scdet scene detection ──► shot boundaries
                                 ├─► ffmpeg frame sampling (one representative frame per shot, capped)
                                 ├─► ffmpeg audio extract ──► ASR transcript (timestamped)   [optional]
                                 └─► OpenAI-compatible vision API ──► evidence JSON

• Scene changes are detected with ffmpeg's scdet filter (ffmpeg ≥ 6.0). Videos without detectable cuts fall back to uniform midpoint sampling.
• ASR is strictly additive: if asrApiKeyEnv is unset or the provider fails, the visual analysis still completes and transcript is null.
• All media work is delegated to ffmpeg/ffprobe on PATH — no native decoding in the agent.

Install

Prerequisites: Node.js ≥ 20, ffmpeg ≥ 6.0 (recommended) with ffprobe on PATH (brew install ffmpeg / apt install ffmpeg).

Option A — npm (recommended)

# in your DSH profile directory (the one containing package.json)
pnpm add dsh-video-lens

Option B — from source (development)

Clone the repo, then mount it into your DSH profile via a local link:

git clone https://github.com/dundunhan/dsh-video-lens.git

Either way, register the bundle in your profile's package.json — this exact block is the full profile configuration:

{
  "dependencies": {
    "dsh-video-lens": "^0.3"
  },
  "dsh": {
    "profile": {
      "bundles": [
        "@deepseek-ai/dsh-base",
        "@deepseek-ai/dsh-web-app",
        "dsh-video-lens"
      ]
    }
  }
}

Then export the keys and restart the profile:

export VIDEO_LENS_API_KEY=sk-...        # vision
export VIDEO_LENS_ASR_KEY=sk-...        # optional, ASR

Configuration

All options are DSH config values:

…