Probes guide — measure what models actually do

Model providers declare capabilities; the probe suite measures them. Four probe classes, each producing dated, versioned JSON results with raw per-sample records — so anyone can recompute the stats and rerun the suite with their own key.

The four capability classes

Class	Question it answers	What a sample records
`schema`	Does the model produce output that parses, conforms to a JSON Schema, and gets the values right?	parse ok, schema conformance, value accuracy vs known-correct fixtures, serving rung (native strict vs json-mode coaching)
`grounding`	Does the model actually search when asked, or fabricate?	citation count (distinct URLs), grounded verdict
`caching`	Is declared prompt caching realized?	cached tokens reported on an identical repeat call
`image`	Can the model read a form image and return schema-conformant JSON extracting the planted fields?	parse ok, schema conformance, value accuracy vs the image's planted fields, serving rung

The three schema levels are deliberately separate: plenty of models return JSON that parses but violates the schema, or conforms to the schema with wrong values. Routing on "supports JSON mode" alone is how silent quality rot happens.

What each measured column means

Each measured leaderboard column links to its method here. Every rate is a sampled statistic with a 95% confidence interval — see Honest stochasticity for what "reproducible" means.

Schema conformance

The share of samples whose output both parsed as JSON and validated against the requested JSON Schema. Parsing and conformance are scored separately: output can parse and still violate the schema. Each sample records parse-ok, schema conformance, and the serving rung (native strict enforcement vs json-mode coaching).

Value accuracy

Among conformant samples, the share whose field values matched the known-correct answer in the fixture. Conformance checks the shape of the output; value accuracy checks the numbers inside it — a model can pass one and fail the other.

Grounded

The share of grounding-class samples that carried at least one citation, counted as distinct URLs. Measures whether the model actually searched when asked rather than answering from memory.

Cache hits

The share of identical repeat calls that reported cached input tokens. Measures whether declared prompt caching is realized on the serving path.

Image input

The share of image-class samples that both conformed to the requested JSON Schema and recovered every planted field from a synthetic form image — a schema-conformant extraction, not just a schema-conformant shape. A response with the right shape but wrong values is not an extraction and does not count. This column is probed-only: a model with no image probe is shown blank, which means unprobed — never "cannot read images". A provider's supports_vision declaration can seed the probe candidate list, but it never fills this column; only a probe does.

Running probes


npx modelrig-probes list-fixtures
npx modelrig-probes run --model openai/gpt-5.2 --class schema --samples 5
npx modelrig-probes run --model deepseek/deepseek-chat --class grounding
npx modelrig-probes run --model openai/gpt-5.2 --class image --samples 5

Results land in results/<provider>-<model>/<class>-<date>.json carrying the harness version, a hash of every fixture exercised, all raw samples, and per-fixture stats. Provider keys come from your environment (OPENAI_API_KEY, GEMINI_API_KEY, DEEPSEEK_API_KEY).

Budget guard: every run carries an envelope (default $5/model, --envelope to change). The run stops with a typed error the moment accrued spend crosses it — partial results are kept and written.

Honest stochasticity

Probes on live models are sampled statistics, not deterministic tests. Results record sample counts, rates, and a Wilson 95% confidence interval — never single-shot verdicts. "Reproducible" means: an independent rerun with the same harness and fixtures lands within the recorded interval.


# integrity: recompute stats from the raw samples in a result file
npx modelrig-probes verify results/openai-gpt-5.2/schema-2026-08-01.json

# reproducibility: compare a fresh run against a published result
npx modelrig-probes run --model openai/gpt-5.2 --class schema --out /tmp/fresh
npx modelrig-probes verify /tmp/fresh/openai-gpt-5.2/schema-<date>.json \
  --against results/openai-gpt-5.2/schema-<date>.json

verify fails loudly if the stats block doesn't match the raw records (tampering) or if fixture hashes differ (you're not running the same inputs).

How results feed the registry

The registry's probed layer takes the latest result per model×class and summarizes it next to the declared layer (vendor claims from the pricing map + adapter capability maps). Where they disagree, the registry entry carries a discrepancies[] array and the leaderboard flags it. Current discrepancy detectors:

declared_schema_low_conformance — claims schema support, probes below 90%
undeclared_schema_capable — no claim, but probes conform ≥90%
schema_served_via_json_mode — claims schema support, but every sample was served by prompt coaching (no native strict enforcement on this path)
declared_search_ungrounded / citations_without_declared_search
declared_caching_unrealized — claims prompt caching, zero cached tokens observed on repeat calls

Hard tier

Fixtures carry an optional difficulty grade: standard (the default) or hard. It is metadata — excluded from the fixture hash and orthogonal to the fixture family, so a probe-suite fixture and a demo-rig fixture can each be hard — and results roll up per tier (by_difficulty) exactly the way they roll up per family. The leaderboard's hard columns read that rollup.

What makes a fixture hard is deliberate. Each hard fixture pulls two or three of five design levers, chosen because they are where capable models genuinely differ and where json-mode coaching diverges from native strict enforcement:

1. Constraint-strict schemas — minItems/maxItems, regex pattern, format: date-time, numeric bounds, enums with plausible distractors, deep additionalProperties: false. 2. Adversarial additionalProperties: false — input that dangles a plausible-but-forbidden field. 3. Value accuracy that needs reasoning, not copying — unit conversion, date arithmetic, cross-field consistency (a total that must reconcile). 4. Instruction robustness — negations, a conflicting secondary instruction, long distractor context before the real task. 5. Structural depth — deeply nested required objects and arrays with per-item required keys.

The standard tier stays comparable across cycles; the hard tier is where strong models separate. Stated plainly: the committed hard set is still small and weighted toward the reasoning lever — arithmetic-consistency traps are what discriminated between strong models in practice — and it broadens with the corpus-expansion cycles.

Coverage honesty

The v0 corpus is small and finance-weighted — it was seeded from our first production customer, which bought realism at the cost of domain breadth. Every fixture carries a domain tag (finance, legal, technical, e-commerce, general, …), the leaderboard states the corpus mix on every render, and per-fixture stats in each result file make per-domain rates derivable. Treat aggregate rates as corpus-wide, never domain-general.

Contributing fixtures

The headline ask is fixtures from your domain. If a model's behavior on your schema shape — or in your subject domain — isn't covered, add a fixture: see the probe kit (kits/probe-kit/) in the public repo. Synthetic data only; set the domain field; provenance is recorded in source and credit lands on the leaderboard.