ModelRig — the operating layer for AI workstreams

These docs are as of modelrig v0.15.0 (the published npm package).

See. Grade. Improve. Point your code at ModelRig to see every model call, grade what's working, and improve cost and quality on the ones that matter — while the registry keeps the routing honest: providers declare capabilities, ModelRig measures them, and a probed fact beats a declared claim.

• See — every run lands in an inspectable artifact store (runs → steps → artifacts, lineaged and hashed), with the cost on each and a place to attach grades. When someone asks why the pipeline did that, you pull up the run.
• Grade — attach a 0–1 grade from your code, a human or a model, and see what's working and what isn't.
• Improve — replay your own recent traffic against cheaper candidates in a bake-off for the effective cost of conformance, with confidence intervals. A winning candidate is a proposal. Never a silent swap: automated changes (opt-in) are proven before they apply, and a change you promote in ModelRig (console or MCP) is versioned and reversible — or merge it as a PR if you keep routes in git.

Start here:

• No account yet? Start free — create your org.
• Quickstart — mint a key, then point your coding agent at it or set it up by hand, raw lane first (no routes needed); your first run on Runs (/runs) in minutes.
• Migration playbook — the ladder: start with runs (no routes), then graduate call sites to routes; the gateway is optional.
• Observe a pipeline — see your existing runs first, without migrating any routes.
• Choosing a lane — observe, the runRaw seam, or named routes: which fits your code.
• Not captured — the calls ModelRig does not see, and how to report one your code makes itself with rig.recordCall.
• Console empty? — the five checks, starting with which org your key exports to (Orgs and keys).
• Route bundles — the routes-as-code reference.
• Probes — the measurement harness: schema conformance (parses ≠ conforms ≠ values-correct), grounding, caching, image input.
• Leaderboard — models ranked by conformance per dollar, declared-vs-probed discrepancies flagged.

Pricing, standard terms: bring your own provider keys (they stay in your environment; the first 1M requests each month are free, then a flat 2% of list price), or route on managed keys (list price + 2%, drawn from your prepaid balance). Same flat 2% either way.

Reproduce any number

npx modelrig-probes run --model deepseek/deepseek-chat --class schema

Every published result carries raw samples, fixture hashes, and a 95% confidence interval. A reproduction passes when your rerun lands inside the interval. Disagreements are contributions: a rerun that lands outside the interval is evidence, so keep its raw samples and fixture hashes.