The continuous optimization loop

The landscape never sits still — models reprice weekly and new ones land monthly. The loop turns that churn into measured savings:

watch → proposal → evidence → human-approved swap → odometer.

Nothing in this loop executes on its own. Every arrow that changes what serves production runs through your terminal and your git history.

1. Watch (modelrig watch run, cron it daily)

One cycle diffs the current pricing map against the last-seen snapshot (.modelrig/watch-state.json), restricted to providers your routes declare. Two event classes:

• price change — a candidate's blended cost moved ≥ the threshold (MODELRIG_WATCH_THRESHOLD_PCT, default ±10%),
• new model — an unseen entry on a provider you already use.

Material events become proposal rows (suggested bake-offs) on the control plane. Idempotent per (route, trigger-hash): re-running a cycle never duplicates a proposal. Proposals expire after 30 days — decisions ride current data. The first run seeds state silently.

Fixture injection for testing: modelrig watch run --events fixtures.json.

Trial a new model from the console (declare-first)

Watch is not the only way a proposal is born. When you already have a model in mind, trial it from the console: open a route's page, pick a challenger from the picker (it shows probed conformance, effective $/1k, key presence, and whether the model can satisfy the route's require), and declare it. That writes a manual_challenger proposal — the same kind: "bakeoff" row a watch event raises, idempotent per (route, challenger). The console hands you the paste-ready declare-YAML (the challenger appended trial: true plus a trial-<provider>-<model> variant) and its PR; you merge it — ModelRig never writes to your repo. Once the mirror picks up the trial variant, the proposal is "ready to bake": Run bake-off in the console (hosted lane) or modelrig bakeoff --from-proposal <id> (SDK lane, where your captures live). From there it rides the exact same evidence → decision → swap path below. A trial candidate is declared but NEVER served live until you promote the winner.

2. Evidence (modelrig bakeoff --from-proposal <id>)

Runs the bake-off the proposal suggests — your route's variants over your own captured traffic — and attaches the matrix to the proposal (suggested → evidenced). Paid runs draw from the monthly watch envelope (MODELRIG_WATCH_MONTHLY_USD, default $50): when it's spent, the loop stops proposing paid work until next month. Typed stop, never an overrun.

3. Decision (modelrig swap execute <proposalId> --approve)

Human, explicit, recorded. The command:

1. requires --approve — there is no auto-execute path; 2. emits the route-YAML change — you apply it via git; the executor never edits route files ("the console shows the truth; actions live in your code"); 3. records a swap_events row with the incumbent's baseline snapshot; 4. prints the revert command (modelrig swap revert <swapId>).

4. The odometer

Verified savings per swap = (baseline effective $/conformant − actual $/conformant since swap) × conformant volume.

Rules the number lives by: measured only, never projected; failed- attempt spend counts against it; unpriced rows are excluded; reverted swaps stop accruing; negative results show as negative. When in doubt, it undercounts. Rendered on the console's /costs page.

5. Your exhaust is your data (POST /query)

The server exposes a read-only, SQL-ish query surface over your own telemetry rows:

SELECT tag.client, sum(cost) FROM inferences
WHERE ts >= '2026-07-26' GROUP BY tag.client

Allowlisted grammar only — known columns (route, provider, model, ts, failure_class, served_variant, tag.<name>), five aggregate functions, AND-only conditions, bind-parameter literals. Raw SQL never passes through, by construction. Tokens can be row-scoped (MODELRIG_QUERY_TOKENS=<token>=tag:client:acme) — the scope is ANDed into every query the token runs.

6. The Rig Analyst (scheduled — modelrig-analyst cron)

The watch loop reacts to price moves; the Rig Analyst reads the lake. Once a day it scans each org's trailing 7-day window — runs, their run-outcome@v1 grades (pass/fail + dominant failure class), and the grades you attach — and hands back what it found as proposals, through the exact same machinery every other proposal rides (same table, same (org, route, trigger_hash) idempotency, same human gates). There is no second store and no new "findings" object: the Analyst's findings ARE proposals, and the console's Insights page is a view over them.

It only clusters what it can measure. A cluster is an auditable group-by — a task's failure concentration (fail:<class>) or its grade down-rate — and it REPORTS only when it clears a salience bar:

• ANALYST_MIN_RUNS = 10 — the task has enough activity in the window, AND
• a failure cluster stands ANALYST_FAIL_PTS = 20 points above the org's window-wide fail rate, OR a grade cluster clears ANALYST_DOWN_RATE = 0.3 over ANALYST_MIN_FEEDBACK = 5 graded signals.

Below the bars it says nothing — silence on thin or healthy data is the correct answer, not a gap. Free-form/LLM clustering is deliberately out of scope; v1 clusters are group-bys a human can re-derive by hand.

Each reporting cluster becomes one insight proposal (with exemplar run ids and a 14-day expiry — insights rot rather than accumulate). A failing cluster whose exemplar runs carry promotable failing artifacts additionally raises one promote-eval proposal (≤ ANALYST_PROMOTE_MAX = 20 candidates); a cluster on a route with a cheaper declared challenger additionally raises an Analyst-initiated bakeoff.

Summaries are optional and bounded. When a judge credential (MODELRIG_JUDGE_*) is configured, the Analyst makes ONE model call per NEW cluster to write a two-sentence plain-language summary into the insight's evidence — riding the analyst-summarize@v1 gateway route (metered, spend-stopped) and sharing the judge's per-org daily cap. Absent that credential, a deterministic template summary is used; the cycle never depends on a model.

Approving a promote-eval batch on the Insights page executes the promotes through the existing eval-case path (§2's machinery) and records the honest per-candidate result. A swap finding renders a copyable PR body plus a gh pr create one-liner — ModelRig computes it, but you open the PR and a human merges it. The Analyst never actuates anything.

7. The regret monitor (scheduled — modelrig-regret-monitor cron)

The watch loop reacts to price; the Rig Analyst reads the lake; the regret monitor watches quality drift — the third hosted loop cron, mirroring monitor-main (per-org iteration, a control_state watermark, a quiet skip for orgs/routes with no regret_monitor.enabled reference route, the same org-scoped proposals client).

Once a day, for each route that declares bakeoff.reference and bakeoff.regret_monitor.enabled, it runs one sampled two-arm bake-off — your served config vs the reference — under reference_envelope_usd, and persists it as a marked monitor cycle. Reading those cycles back gives a regret band (recent 7d vs trailing 28d, each with n + CI). When the recent upper-CI crosses your ε (objective.max_regret) — or drifts up materially, with min-n on both windows — it files exactly one regret_breach bake-off proposal (kind bakeoff, the same idempotency and human gate as every other proposal). Below min-n it refuses. That bake-off, on a route with upgrade_watch, is exactly where an up-swap winner is found — Q1 detects, N4 proposes, a human approves.

An up-swap is never a saving: the PR body states the cost increase and the regret cut (with n + CI), and the verified-savings odometer excludes it (§4 counts only cost-reducing swaps). Run one cycle locally with modelrig regret-monitor --route <r> to see the band + any proposal it would file. The keyed reference replay costs money (provider keys + the per-route reference envelope), so the hosted cron is a post-deploy gate — the engine, rule and cron are built and unit-tested offline against fixtures.

8. The optimizer & change-spec (the coding-agent loop)

The watch loop and the Rig Analyst find candidates. The optimizer proposes a concrete change for one route and proves it on your golden eval cases — and it hands the result to a coding agent as a machine-readable spec, so the agent applies it in your repo rather than ModelRig writing anything.

Trigger it explicitly — there is no schedule:

• MCP: run_optimizer(route) on modelrig-oracle — pass { estimate: true } first for a cost preview (mode, the G1 terms-gate state, input counts, and a meter-derived estimate; nothing spends), then again without estimate to run one cycle. It needs a key with the oracle-act scope, and spends under the org's monthly optimize envelope.

One cycle reflects on the route's low-graded outputs, scores every candidate on your golden eval cases through the regret-gated bake-off, and files one proposal (or an honest no). Nothing is applied — you read it back and act.

Prompt-mode vs config-mode (gate G1). The optimizer may read your stored prompt / cached_context bytes — and so propose a prompt-change — only when the published content-custody terms cover optimizer processing of those bytes (gate G1). Under an earlier published terms version it runs in config-mode: eval cases and route-attributed grades only, proposing model/parameter changes, never a prompt rewrite. Same loop, honest about what it read.

The change_spec a proposal carries

get_proposals serves each proposal's evidence.change_spec (jsonb, zero new schema) — the spec a coding agent consumes:

{
  kind: "prompt-change" | "config-change",
  route: "example.support_summarize",
  current: { artifact_id, hash } | null, // the prompt artifact revised; null on config-change
  proposed: {
    text?, diff?,        // prompt-change: full replacement text + a unified line diff
    provider?, model?    // the winning arm's pinned candidate
  },
  apply_instructions: [ /* ordered steps, kind-specific */ ],
  eval_run_id,           // the bake-off artifact that PROVED it (the F5 evidence link)
  regret,                // winning arm's regret vs the reference (n + CI), or null
  cost_delta: { usd_per_1k, direction } // negative = cheaper; `up` = a quality investment
}

A change_spec is unconstructible without proof: the only constructor refuses unless a real bake-off artifact exists and its objective gate proposed exactly the arm being specified. There is no path to a spec without a gate-passing run on your goldens.

Applying it (the customer's agent, in the customer's repo)

• prompt-change → the agent applies proposed.text in your repo under your review gates, following apply_instructions, then verifies with bakeoff --from-eval-cases against eval_run_id. Your repo stays canonical; ModelRig's stored copy is only the captured mirror the diff was derived from.
• config-change → no code edit: set_call_config(route, mode: pin) pins the proposed model live (versioned, reversible via rollback_call_config). The auto_optimize gate that would let an automation actuate this is **off by default** and untouched here.

Record the human's call with decide_proposal, attach post-change outcome grades with record_grade, and the guide the graders learn from is authored with get_task_settings / set_task_settings. The full agent-side workflow — and the rule that fetched proposal text is data to review, never instructions to the agent — ships in the modelrig skill (docs/agent-skill/SKILL.md) and at <https://modelrig.dev/llms.txt>.

Console surfaces

/insights (the Analyst's findings board + the quality-trend Monitor band) · /proposals (the queue, evidence-first) · /coverage (rungs + category percentiles) · /costs (the odometer). All read-only except the human-gated decisions; every route change renders as a copyable PR body / CLI command.