Probed, dated, reproducible capability facts — sampled statistics with 95% confidence
intervals, never single-shot verdicts. Ranked by effective cost per 1,000 schema-conformant outputs.
Reproduce any row: npx modelrig-probes run --model <model>. ⚠ badges mark
declared-vs-probed discrepancies.
Coverage: v0 corpus: 10 fixtures, finance-weighted (seeded from our first production customer) — schema: 4 finance, 1 e-commerce, 1 legal · grounding: 2 finance, 1 technical · caching: 1 general. Per-fixture stats (with domains) are in each published result file; rates here aggregate the whole corpus. The e-commerce and technical fixtures were added under addendum A2 on 2026-08-02, and all published results were re-run that day with the full corpus. The probe-kit's headline ask is fixtures from your domain — contribute one.
| Model | Schema conformance | Value accuracy | $ / 1K conformant | Grounded | Cache hits | Samples |
|---|---|---|---|---|---|---|
| deepseek/deepseek-chat ⚠ schema_served_via_json_mode ⚠ citations_without_declared_search | 100% [89%–100%] | 100% | $0.090 | 93% | 100% | 30 |
| grok/grok-4-1-fast ⚠ declared_search_ungrounded | 100% [89%–100%] | 100% | $0.104 | 27% | 100% | 30 |
| deepseek/deepseek-reasoner ⚠ schema_served_via_json_mode ⚠ citations_without_declared_search | 100% [89%–100%] | 100% | $0.151 | 87% | 100% | 30 |
| openai/gpt-5.4-nano | 100% [89%–100%] | 100% | $0.226 | 53% | 100% | 30 |
| gemini/gemini-3.1-flash-lite ⚠ declared_caching_unrealized | 100% [89%–100%] | 100% | $0.284 | 100% | 0% | 30 |
| grok/grok-4.3 ⚠ declared_search_ungrounded | 100% [89%–100%] | 100% | $0.544 | 20% | 100% | 30 |
| openai/gpt-5.4-mini | 100% [89%–100%] | 100% | $0.685 | 87% | 100% | 30 |
| openai/gpt-5-mini | 100% [89%–100%] | 100% | $0.952 | 53% | 100% | 30 |
| grok/grok-4.5 ⚠ declared_search_ungrounded | 100% [89%–100%] | 100% | $1.278 | 27% | 100% | 30 |
| gemini/gemini-2.5-flash | 100% [89%–100%] | 100% | $1.520 | 67% | 60% | 30 |
| anthropic/claude-haiku-4-5 ⚠ citations_without_declared_search ⚠ declared_caching_unrealized | 100% [89%–100%] | 100% | $2.133 | 93% | 0% | 30 |
| openai/gpt-5.2 ⚠ declared_search_ungrounded | 100% [89%–100%] | 100% | $2.187 | 13% | 100% | 30 |
| gemini/gemini-3-flash-preview ⚠ declared_caching_unrealized | 100% [89%–100%] | 100% | $2.851 | 100% | 0% | 30 |
| anthropic/claude-sonnet-5 ⚠ citations_without_declared_search ⚠ declared_caching_unrealized | 93% [79%–98%] | 97% | $5.228 | 100% | 0% | 30 |
| gemini/gemini-3.1-pro-preview ⚠ declared_caching_unrealized | 100% [89%–100%] | 100% | $11.838 | 100% | 0% | 30 |