OMK

English | 简体中文

Observe. Measure. Know.

OMK makes every knowledge change in your AI application evidence-backed.

Observe real-world performance, measure version differences, and determine whether the change is effective and the version is ready to ship.

Same model. Same evaluation samples. Only the knowledge artifact changes.

DeepSeek Harness users: install OMK as a native bundle, reuse the current profile for controlled evaluations, and open persisted DSH task trajectories in Studio. Set up the DSH host plugin →

📖 Full documentation: oh-my-knowledge.pages.dev (searchable, English / 简体中文)

What OMK helps you know

Decision	Command	Evidence you get
Is this artifact coherent enough to evaluate?	omk doctor	structure, dependencies, safety, and measurability checks
Is v2 actually better than v1?	omk eval	one-line verdict, confidence interval, failed samples, cost
Why did it pass or fail?	omk studio	report view with scores, diagnostics, and examples
Should this version become the accepted one?	omk promote / omk evolve	evidence-gated accept or generate a better candidate
What happened during one real AI task?	omk observe / Studio Task Trajectory	a trace-backed view of the request, visible Knowledge, tool calls, results, response, and user correction
What did real usage expose?	omk observe / omk sample --from-traces	production gaps drafted for review; reviewed drafts can become eval samples

Quick start

npm i -g oh-my-knowledge
omk init demo && cd demo
omk eval --control code-review-v1 --treatment code-review-v2 --dry-run
omk eval --control code-review-v1 --treatment code-review-v2

Runs out of the box — no edits needed first. omk init scaffolds two skill variants and three sample cases; --dry-run previews calls and cost; omk eval runs the controlled A/B and opens an HTML report with a one-line verdict in about five minutes. Once it runs, swap in your own skills and cases.

Prerequisite: configure one authenticated model runtime (Codex CLI, Claude Code, or an API executor; see Requirements). Inside a Codex task in the ChatGPT desktop app, omk automatically selects codex, reads the model from ~/.codex/config.toml, and uses the same Codex model as the default judge. Claude is not required.

To make Codex the default in regular terminals, add the preference to your shell profile (for example ~/.zshrc):

export OMK_EXECUTOR=codex
# Optional: export OMK_MODEL="your-codex-model"

Without OMK_MODEL, omk reads the model from ~/.codex/config.toml. You can still pass --executor codex --model <codex-model> per command. Pass --judge-models or set OMK_JUDGE_MODELS only when you want a different judge.

The first run has only 3 cases, so the verdict will usually be UNDERPOWERED (insufficient data) — that's a normal starting point, not an error; grow to ~20+ cases before trusting a ship/no-ship call.

The CLI notifies you when a newer version is available (at most once per 20h); set OMK_SKIP_UPDATE_CHECK=1 to silence it permanently.

…