Routing & reliability — how a rig.run call actually behaves

rig.run(route, { input, tags }) returns a validated output plus a meta record — a route is a contract, not a single model call. This page is about the candidate ladder: the runtime loop that decides which model serves the call, what makes it fall to the next one, what is stored along the way, and — just as important — what it does not do. It is a different thing from the migration playbook's autonomy ladder (the T0–T3 tiers), which happens to share the word "ladder".

The vocabulary this page fixes, used the same way everywhere: a candidate ladder is the runtime loop; an attempt is one dispatch, which is one telemetry row; fall-through is advancing to the next candidate; the repair rung is the schema-repair step; a failure class is the typed reason an attempt failed; the conformance gate is what the ladder enforces (schema). A quality gate (a value judgment) is available on the SDK lane as a builder-supplied qualityGate predicate — it rejects a schema-valid output as quality_rejected and falls the ladder through (§4). The route-DECLARED judge that would do the same on the hosted lane is still not built (§10).

1. Resolve — who is eligible, and in what order

Before any model is called, resolve builds the eligible candidate list and orders it. Filters apply in a fixed order: providers.only (keep only these providers) → providers.ignore (drop these) → require (the hard capability constraints, including a per-run zeroRetention opt-in) → maxPriceUsdPerMTok (a price ceiling — an unpriced candidate is dropped when a ceiling is set, fail closed) → sort.

require names hard constraints, and each one filters candidates:

• schema_conformant with json: native keeps only a candidate whose resolved flags include structured_native (overrides > probed > declared); with json_mode it keeps either structured_native or json_mode.
• grounded keeps native-grounded candidates, and a non-native candidate only if a search provider is configured (the grounding-inject rung); otherwise that candidate is dropped.
• trace_visible keeps candidates that expose the reasoning trace to the caller.
• zero_retention routes only to zero-retention–designated endpoints and fails closed — no designated candidate means the no-eligible-candidate error, never a silent non-ZDR dispatch. A single run opts in with `RunOptions.zeroRetention: true` (opt-in only, never opt-out).

Capability flags resolve by precedence: overrides > probed > declared — a probed-false beats a declared-true.

Ordering: prefer: [cost] sorts by blended (in+out) $/MTok ascending; an unknown price sorts last; the sort is stable, so ties keep declared order. prefer: latency is accepted and a no-op today — declared order stands. No prefer means declared order.

If nothing is eligible, resolve raises RouteConfigError before any dispatch — there is no telemetry row. (A route with zero serveable candidates fails earlier still, at load: createRig fails.)

A keyless candidate — one whose provider key is absent in this environment — is skipped at run time: no dispatch, no row, no spend. It is remembered as a config_auth last-failure, so a ladder of only keyless candidates throws config_auth.

A trial candidate — one declared trial: true — is dropped from the pool here, on every live path, before ordering. It is not keyless and not ineligible; it is declared but deliberately never served live. The only place a trial candidate is kept is a bake-off replay, where it is measured against the incumbent on your captured traffic; selecting a trial-only variant on a live call raises RouteConfigError naming the bake-off, never a silent serve. This is how you evaluate a new model before promoting it: declare it trial, bake it off, and only then move it to a live candidate. (Deploy a build that understands trial: before any YAML carries it — an older resolver has no trial filter and would serve it live.)

2. The candidate ladder

resolve -> ordered candidate list
for each candidate, in order:
  render prompt (variant prompt_append / scaffold)
    + json-mode coaching pasted in for any candidate lacking structured_native
    + grounding search: runs ONCE per run, reused across candidates and retries
    -> candidate-set invariant check
    -> envelope precheck (checks an upper-bound estimate against the envelope)
    -> dispatch on YOUR key
         |
         +- tool-call turn?          -> return output=null, meta.toolCalls (NO validation)
         +- valid?                   -> record + capture (if on) + return
         +- schema-invalid?          -> content_invalid row -> repair rung
         |                              -> retry SAME candidate while its budget lasts
         |                              -> then the next candidate
         +- retryable infra class?   -> retry SAME candidate within that class's
         |                              budget (with backoff) -> then the next candidate
         +- config_auth?             -> the next candidate, immediately
         +- budget_exhausted /
            invariant_violation?     -> ABORT the run

ladder exhausted -> RigFailureError(LAST failure)     # there is no meta on a failure

Per candidate, in order: the prompt is rendered (variant prompt_append or scaffold); json-mode coaching is pasted into the prompt for any candidate lacking structured_native (and always on json: json_mode); the grounding search runs once per run and is reused across candidates and retries (a search failure is a network failure); a candidate-set invariant check runs before adapter lookup; the envelope precheck checks an upper-bound estimate against the envelope; then the model is dispatched on your key.

The outcomes:

• A tool-call turn short-circuits before validation: output is null, meta.toolCalls is set, meta.validated is false — ModelRig never runs the tool (§9, and boundary B9).
• A valid output is recorded, captured (when capture is on), and returned.
• A schema-invalid output writes a content_invalid row, then the repair rung runs; the loop then retries the same candidate while its content_invalid budget has units (repair itself does not re-run), and only then advances.
• A retryable infrastructure class retries the same candidate within that class's budget (with backoff), then advances.
• config_auth advances to the next candidate immediately.
• budget_exhausted or invariant_violation aborts the run.

When the ladder is exhausted, rig.run throws RigFailureError carrying the last failure — .failure.class, .failure.message, an optional .failure.provider / .model / .fixHint, and a .docsUrl on the teaching render — not a generic "all candidates exhausted" message (that string appears only if nothing at all was recorded). There is no meta on a failure.

3. The repair rung, exactly

Repair is off unless a repair: block is declared. max_repairs is 1 or 2; 2 requires repair_model, which must be one of the route's declared candidates. Attempt 1 re-asks the same model with the ajv error summary appended; attempt 2 hands {invalid output, errors, schema} — never the task prompt — to the repair model. The repair budget is its own (never the content_invalid budget) and is created once per run — it does not re-arm for the next candidate. Repair rows carry repaired_by. A variant may override the repair block, but the effective repair policy is fixed for the run — the budget is still created once, per run, not per candidate.

4. What makes the ladder fall to the next model — and what does not

The ladder has an OPTIONAL quality gate on the SDK lane: pass rig.run(route, { qualityGate }) and a schema-valid output the predicate rejects becomes a quality_rejected attempt on its own retry budget (default 0), falling through without ever entering the repair rung. Without a gate, a schema-valid answer is accepted as before. The same predicate scores value accuracy offline, in bake-offs — see the boundary (§10, the route-declared hosted judge is not built) and how bake-offs measure it.

5. Failure classes

Every failure is typed. Default retry budget for every class is 0 — a bundle declares each budget it wants under policy.retries. Each class's docsUrl on the rendered RigFailureError points at its heading below, so the thrown error links to exactly the section that explains it.

content_invalid

The model answered, but the answer did not match the schema your route requires — so it was rejected rather than passed downstream.

Retryable — default budget 0 (declare every budget you want under policy.retries).

No backoff between retries (immediate retry).

Fix: Retry on this class is budgeted separately, so a flaky schema cannot eat your network retries. If it persists, the model likely cannot hold this shape: check the registry for a candidate with probed native structured output.

capacity_shed

The provider shed your request rather than serving it — a 429, a 503, or a flex-tier rejection. Nothing was wrong with the request.

Retryable — default budget 0 (declare every budget you want under policy.retries).

Backoff between retries: 30 s -> 60 s -> 120 s -> 240 s.

Fix: Capacity retries back off automatically. If it happens constantly, you are asking for discounted sheddable capacity during peak: move the route off the flex tier, or add a candidate on another provider so a shed on one does not stall the route.

network

The request never completed at the transport level — DNS, TLS, a dropped connection, or an error we could not classify any more precisely.

Retryable — default budget 0 (declare every budget you want under policy.retries).

Backoff between retries: 1 s, doubling each time, capped at 30 s.

Fix: These retry on their own budget. A persistent one is usually local: check egress, proxy settings, and whether the provider is up before changing anything in the route.

refusal

The model declined the task itself — this is an answer, not a fault.

Retryable — default budget 0 (declare every budget you want under policy.retries).

No backoff between retries (immediate retry).

Fix: Retrying a refusal usually earns another refusal. Read the prompt as the model saw it: the input often contains something the model's policy will not touch. A different candidate may answer, but check the prompt first.

cache_invalid

The provider-side cache or cached context this call relied on expired or was rejected, so the discount it was counting on is gone.

Retryable — default budget 0 (declare every budget you want under policy.retries).

No backoff between retries (immediate retry).

Fix: Re-running usually repopulates the cache. If it recurs, your cached prefix is probably changing between calls — anything varying near the START of the prompt defeats prefix caching, so move volatile values to the end.

timeout

The call passed the route's wall-clock deadline before the model finished.

Retryable — default budget 0 (declare every budget you want under policy.retries).

No backoff between retries (immediate retry).

Fix: Either the deadline is too tight for this task or the model is slower than assumed. Check the latency column for this model before raising the timeout — a timeout raised to cover a slow model just makes a slow route.

config_auth

The provider rejected the credentials, or there were none to send. This is never retried — retrying a dead key just spends time.

Never retried — the run loop advances to the next candidate.

Fix: Set the provider's API key in this environment. Until then the candidate is inert: the route still serves on its keyed candidates, and this one is skipped at dispatch.

budget_exhausted

The envelope this work was running under hit its cap, so spending stopped mid-run. This is the budget machinery working, not a bug.

Terminal — never retried; aborts the run.

Fix: Decide whether the cap was wrong or the run was. Check the envelope's spend against what you expected before raising it — a runaway loop and an under-sized cap look identical from here, and only one of them is fixed by more money.

invariant_violation

Something tried to serve a model your route never declared. The run was stopped rather than allowed to bill you for a model you did not choose.

Terminal — never retried; aborts the run.

Fix: This is a bug in ModelRig, not in your configuration — the candidate set is meant to be an upper bound that nothing can escape. Please report it with the route name and the model that appeared.

quality_rejected

The output met the schema your route requires, but the quality gate you supplied (rig.run(route, { qualityGate })) judged it below your bar and rejected it — so the ladder fell through rather than passing it downstream.

Retryable — default budget 0 (declare every budget you want under policy.retries).

No backoff between retries (immediate retry).

Fix: This is your own predicate talking, not the model failing a shape check. If good outputs are being rejected, the bar is too tight — loosen the gate. If you want the same candidate re-sampled before falling through, give this class a budget under policy.retries; it defaults to 0 (immediate fall-through) and never enters the repair rung.

6. What is stored, per attempt — and where it goes

One telemetry row is written per attempt — a serving attempt, a failed attempt, and a repair attempt each get their own. The row carries: id (this is meta.inferenceId), route and version, provider and model, requested and served tier, tokens in / out / cached (plus cache-write), cost, latencyMs, ttfbMs, failureClass (null means conformant — there is no validated column on the row), attempts, tags, servedVariant, repairedBy, the cache key and prefix fingerprint, and pricingMissing. Grounding adds a synthetic provider = "search" cost row.

The meta returned to your code (RunMeta) is a smaller record: inferenceId, route, routeVersion, provider, model, requestedTier, servedTier, tokensIn, tokensOut, tokensCached, costEstimateUsd, latencyMs, validated, attemptsByClass, servedVariant, repairedBy, and — when they apply — citations and toolCalls. (RunMeta.validated is a boolean you read; the stored row instead records conformance as failureClass IS NULL.)

Where the rows go: SQLite first, always. Setting MODELRIG_API_KEY turns on the api export mode — a background exporter ships batches to the ingest API (api.modelrig.ai, override with MODELRIG_INGEST_URL), which the console reads. It is fire-and-forget and never blocks a run. Api mode also opens the capture store, and a control-plane capture setting wins over the YAML capture: field.

Who reads them: bake-offs and replay read captures; rig.grade and the native run grades read the rows. effective_usd_per_1k_conformant is a bake-off metric computed over replayed captures, not a rollup of production rows.

7. Spend envelopes

Envelopes are opt-in per run via budget.envelope. Open-or-get: the first opener of a named envelope sets its budget and it is never resized; the default is $25. A precheck checks an upper-bound estimate against the envelope before dispatch and aborts the run (terminal budget_exhausted) on breach. An unpriced model is charged a conservative rate against the envelope while telemetry records $0 plus pricingMissing. Envelope state persists in SQLite across restarts.

8. The raw lane is different

rig.runRaw calls one provider/model directly with a BYOK apiKey. It has no ladder, no require, no validation, no repair — and no envelope; meta.validated is always false. It writes one lane: "raw"-tagged row on success and on adapter failure, and no row when a knob is rejected pre-dispatch. Which knobs each provider honors is the generated provider × knob matrix — read it there; it is never restated here. Constructing a runRaw-only rig with routesDir: null needs modelrig ≥ 0.4.0; see the migration playbook.

The multi-provider reasoning / native-grounding knobs and the responseFormat knob on the raw lane need modelrig ≥ 0.4.0 (on 0.3.0, reasoning and native grounding are gemini-only and responseFormat does not exist). The provider × knob matrix marks each one.

9. Also on the ladder

• Streaming: rig.runStream is the identical ladder with a delta side-channel; deltas are pre-validation bytes and the final event is authoritative.
• Variants narrow the candidate set and can override the prompt, json, or repair; an unknown variant name is a RouteConfigError (no dispatch); meta.servedVariant names the variant that served (null = default).
• cache: auto stamps the per-provider cache directive (gemini is excluded); a customer cache handle (RunOptions.cache, provider-scoped) wins over the stamp and is dropped on retry after cache_invalid → see caching lifecycle.
• The engine adds tags: a ZDR-effective run gets zdr_enforced: "true"; the raw lane force-stamps lane: "raw".
• Citations are normalized onto meta.citations on the grounding-inject rung.
• Observed latency (p50 / p90 / p99) is aggregated but **does not influence routing** today (§10, B3).

10. Not yet implemented — the boundary

The most useful thing docs can tell you is what the engine will not do, so you size the product correctly. The entries below are grouped by status — not built, deferred, built but not actuating, and by design (will not do).

Not built

Deferred

Built, not actuating

By design — will not do

This list is generated from code and tripwire-tested: if a feature listed here ships, the build fails until the entry is updated.