Bake-offs, replay & capture

"Can a cheaper model do this step?" — answered with measurements, not vibes. A bake-off replays your own captured traffic through named route variants and produces the conformance matrix: per variant, conformance rate with a Wilson 95% interval, repair rate and repair cost, latency, and the headline — effective $ per 1K conformant outputs (total spend ÷ conformant outputs × 1000, including retries, repair calls, and search costs).

1. Capture (local-only — the privacy invariant)

Replay needs inputs. Opt a route in per bundle:


capture: true

Every attempt then writes {rendered variables, output text} to the local SQLite captures table.

Captures never leave the machine. This is structural, not policy: the telemetry exporter's store interface has no capture accessors — there is no code path from captures to any network sink — and a runtime export-isolation test asserts no sink payload ever carries capture content. The cloud schema deliberately has no captures table. What does mirror: the per-route capture flag (status only) and bake-off aggregate metrics — and, when a bake-off is run with --keep-outputs, the bounded review samples it keeps, which land in bakeoff_samples in your organization, row-scoped by the database and deletable from the console. Turning capture: true on a route is what the brandscript calls Optimization on; leaving it off is Pure router.

2. Replay


const samples = await rig.replay("example.support_summarize", {
  variant: "cheap",
  lastN: 50,
  envelopeUsd: 10, // hard cap for the whole pass
});

Replay re-renders the captured variables through the variant's template against the replay candidate's capability flags (never the captured prompt text — a variant with different capabilities must render as it would serve). Replay calls are real, billed calls: envelope-guarded, tagged into the replay:<id> tag namespace (production tag queries never see them), and never written back to captures.

3. Bake-off


modelrig bakeoff --route example.support_summarize \
  --variants default,cheap,scaffolded --replay-last 50 --budget-usd 5

--budget-usd <n> sets the spend envelope per replay variant (decision 3 — an explicit cost ceiling on a bake-off). When a variant's replay pass breaches it, the pass stops typed at that point: the breaching sample is recorded with failureClass: budget_exhausted and the partial result set stands and is reported as partial — a bake-off never silently spends past its envelope. Omitted ⇒ a sane default cap. (--envelope is a kept alias for the same knob.)

default is the route's base config. Output: the matrix on stdout, an artifact JSON under .modelrig/bakeoffs/, and (when the Supabase sink is configured) a row in the bakeoffs mirror rendered by the console's /bakeoffs view. Also available over HTTP: POST /bakeoff/:route returns 202 + a poll URL (GET /bakeoff/:id).

4. Proposals are proposals

A variant proposes when it clears the route's objective (§5). Under cost-first that is: conformance CI lower bound ≥ the incumbent's and effective cost strictly lower; when the bake-off carries a qualityGate, a third clause applies — the variant's quality-pass CI lower bound must also be ≥ the incumbent's — and the cost metric becomes effectiveUsdPer1kPassing (reported beside, never instead of, effectiveUsdPer1kConformant). The artifact records the winner and the savings percentage — and that is all it does. Model swaps in production are a human decision: nothing in the bake-off path mutates routing; a route-declared judge on the hosted lane is not built.

5. Near-optimal, not cheapest — the objective

The regret-bounded objective (objective, bakeoff.reference, value_checks, bakeoff.judge) and the quality-first / balanced gates below ship in 0.6.0. A route that declares NEITHER an objective: NOR a bakeoff: block keeps the cost-first gate, byte-identical.

For a route that opts in, the bake-off objective is no longer "cheapest that passes." It is near-optimal at a reasonable price: minimize the served model's quality regret vs the best available model on your tasks, subject to cost ≤ a ceiling you set. Three quantities make that operational, all declared on the route (additive YAML):


objective:
  mode: quality-first        # default; also: balanced | cost-first
  max_regret: 0.02           # ε — the quality-points gap you tolerate vs the reference
  price_ceiling: { usd_per_1k: 4.00 }   # or { usd_per_run: 0.05 }
  lambda: 0.5                # balanced only: minimize regret + λ·cost_norm
bakeoff:
  reference: auto            # the ceiling arm — or provider/model explicitly
  reference_envelope_usd: 2.00          # the reference is dear by definition
value_checks:                # deterministic value accuracy (no judge needed)
  - { path: /cagr, expect: { regex: "^[0-9.]+%$" } }

Reference (ceiling) arm. Every bake-off can run a reference on the same replayed samples — by default the strongest declared candidate for the task (by probed schema conformance in the registry; reference: auto), or one you name. Each other arm reports regret vs the reference, with a CI and n.
Value accuracy by default. Quality per sample resolves, in order: an eval-case golden → declared value_checks (JSON-pointer assertions) → an opt-in capped judge (bakeoff.judge: { model, max_samples }; the judge model must be a declared candidate) → else 1/0 conformance. The report NAMES the metric it used (value_accuracy or conformance).
The gate. quality-first proposes the lowest-regret arm whose regret CI upper bound ≤ max_regret and whose cost ≤ the ceiling (n_paired ≥ 10), and only when it beats the incumbent on regret or cost — a better-and-dearer model is proposable while it stays under your ceiling. cost-first reproduces the older gate byte-for-byte. **No bound declared ⇒ regret is reported but nothing is proposed** — the default flip never actuates silently.


modelrig bakeoff --route example.support_summarize --variants default,cheap \
  --replay-last 200 --objective quality-first

6. Watching the ceiling — up-swaps and the regret monitor

A bake-off is a snapshot. Two things drift after it: the ceiling rises (a new model, or a new snapshot of your reference, raises what's achievable on your fixtures), and your regret grows silently (the served model gets nerfed, traffic shifts, a better model ships). N4 and Q1 make both standing signals.

Up-swaps (N4). RB-1's quality-first gate already returns a better-and-dearer winner under your ceiling. N4 adds the trigger — when your reference/ceiling rises, the watch loop files an upgrade_opportunity bake-off (the mirror of a price move) — and the direction label: at evidence time each proposal is stamped swap_direction: up | down | lateral from sign(winner − incumbent) on the effective-$/1k axis, plus the signed cost delta. An up-swap raises your cost to cut regret, so it is treated honestly everywhere: the PR body says "raises effective cost by \$X/1k to cut regret Y pts (CI …, n=…) — a quality investment, not a saving", the --approve prompt shows the cost going up before you say yes, and the verified-savings odometer excludes it entirely (an up-swap moves the savings headline by exactly \$0, while staying in the swap history with its ⬆ label). Up-swaps never auto-execute — a human approves the PR, exactly like a down-swap.

The regret monitor (Q1). A scheduled, envelope-bounded cycle keeps regret measured over time. Each cycle is a two-arm sampled bake-off — your served config vs the route's reference — under reference_envelope_usd; it reuses the same engine, so the reference arm, paired regret, partial-result honesty and metering all come for free (replay is tagged lane: trial and never moves a live odometer). Cycles mirror as marked bakeoffs rows, so a regret band reads back: a recent 7-day window vs a trailing 28-day baseline, each with its n and CI, and a drift arrow. When the recent upper-CI crosses your ε (max_regret), or drifts up materially — with enough paired samples on both windows to trust the number — it files exactly one regret_breach bake-off proposal. Below min-n it refuses (measured-only). Q1 detects → N4 proposes → a human approves.


# modelrig/routes/example.support_summarize.yaml
objective:
  mode: quality-first
  max_regret: 0.02            # ε — the regret bound the monitor watches
bakeoff:
  reference: auto             # required for both — nothing to measure against otherwise
  reference_envelope_usd: 2.00
  regret_monitor: { enabled: true, sample_n: 200 }   # Q1
  upgrade_watch: true         # N4 (a route with a reference defaults this on)


modelrig regret-monitor --route example.support_summarize   # one cycle: band + any breach

Honest limits (claim ceiling)

Near-optimal is always measured, never asserted. Sanctioned phrasings: "within 0.8 pts of the reference on 212 replayed runs (CI shown)"; "near-optimal — measured, dated"; "better-and-dearer proposed only under your ceiling". Never use an unqualified superlative for a model's quality, never state a regret number without its n and CI on the same surface, and never promise "guaranteed quality" (enforced by the www claim-lint). reference: auto ranks by the registry's probed schema conformance — a measured signal, not a frontier oracle — so a route with no probed candidate resolves no reference (and no regret) rather than guessing.