The ModelRig grade protocol (open)

A grade is how judgment enters the optimization loop: a score in [0, 1] — with an optional written comment — that you attach to a subject (one model call, a whole run, or an artifact). The judgment can come from anywhere: your own code, a human, or an AI. ModelRig aggregates grades into per-route optimization coverage — the evidence level that decides how much confidence a swap proposal carries. The protocol is deliberately open: any tool that emits rows in this shape can participate in the loop.

The score is the one primitive. The pass/fail verdict is derived, never stored — pass = score ≥ the route's threshold (default 0.5; see the derived verdict). A thumbs-up is just a score of 1, a thumbs-down a 0; a rubric grade like "88 / 100" rides the optional display scale. There is no scoreless judgment.

Every grade is org-scoped — it lands in the organization its credential names — and a grade attached to a run renders on that run's page in the console (see on the run page).

The subject

A grade names exactly one subject by kind and id:

Attribution is never guessed: a grade whose subject matches no known inference, run, or artifact attaches to nothing.

The call

// one call — the inference id rides on every RunResult:
const result = await rig.run("example.support_summarize", { input, tags: { run_id } });
rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" });

// a whole run (the subject id is your run_id tag):
rig.grade(
  { kind: "run", id: run_id },
  { score: 0.2, comment: "CAGR off by 10x", kind: "human", source: "review-agent" }
);

The write is synchronous and local (SQLite) — never a network call, never a failure your path can see. The cloud mirror rides the telemetry exporter's normal cadence. To post a grade from a surface that isn't running the SDK — an end user clicking thumbs-down in your web app — use the HTTP endpoint below.

The grade

rig.grade(subject, {
  score: 0.88,                    // 0–1 PRIMITIVE (always present)
  scale: { max: 100 },            // optional display only — { max } or { label: "B+" }
  comment: "Weigh the pending competition next time.",   // written guidance
  kind: "human",                  // "human" | "model-judge" | "deterministic"
  grader: "gpt-5-judge",          // optional specific grader name
  source: "review-agent",         // provenance (sdk, console, an agent name…)
});

The three grader kinds

A grade's kind records what did the judging, and the three are kept apart on purpose:

• deterministic — a recomputable check with no model call (a schema conformance pass, a value assertion). Same inputs, same grade, every time.
• model-judge — a model scored it. Costs a call, so it is always explicit and metered (see run-judge@v1).
• human — a person's judgment, or a signal you already have (a downstream accept/reject, a QA gate) recorded as one.

The derived verdict (per route)

The pass/fail verdict is computed at read time, never written: pass = score ≥ routeThreshold(subject's route). Each route may set its own threshold in its grade policy (grade.pass_threshold, 0–1, default 0.5); a subject with no route — a raw-lane call — uses 0.5. Coverage, insights, and the console's pass badge all derive the verdict from the one stored score, so they never disagree.

Native run grades

A single model call either conformed to its schema or it didn't; a whole run — a pipeline of steps and the artifacts they produced — needs a grade of its own. ModelRig writes two kinds automatically, and keeps them honest by keeping them apart: one is computed, free, and automatic; the other calls a model, costs money, and never fires on its own.

run-outcome@v1 — deterministic, automatic

When a run ends, ModelRig writes one run-outcome@v1 grade with no model call — a deterministic grade on the run subject. It scores 1.0 (a pass) iff all three hold:

1. the run succeeded (a running, failed, or abandoned run never passes); 2. no step's latest attempt carries a failure_class; 3. every deterministic grade in the run passed.

Otherwise it scores 0.0, and the grade's display label names the dominant failure — the most frequent step failure class, else the run's non-succeeded status, else failed-evaluation. Same inputs, same grade, every time: it is a pure function over columns the run graph already carries, so it costs nothing and never disagrees with itself. This is the measured floor — a grade you have before any judge is involved.

run-judge@v1 — a model judge, explicit and bounded

Some questions ("did this run actually achieve the outcome?") a deterministic check can't answer. For those there is one built-in model judge, run-judge@v1: it reads the run's step/artifact summary (metadata, declared schemas, and the deterministic grades; content only where custody grants it) and scores outcome-achievement 0–1 with a comment, written as a model-judge grade.

It is deliberately fenced in:

• Explicit only. It runs when you ask — the console's judge button (admin/owner) or modelrig judge <runId> — and never on a schedule or automatically. There is no background judging.
• Metered and routed. The judge is a route: the scoring call rides the gateway like any other, so it is priced, budgeted, and spend-stopped the same way, and every invocation is a counted, metered event. It honours the route's pass threshold.
• Capped per org, per day. A conservative daily cap bounds spend; past it the call refuses with a typed error (429) rather than run up a bill.

modelrig judge run-123      # score one run with the bounded judge; explicit + metered

Attach your own

Your own code attaches a run-level (or artifact-level) grade the same way as any other — rig.grade with the subject kind:

rig.grade(
  { kind: "run", id: run_id },
  {
    score: 0.9,
    comment: "all figures tied out",
    kind: "deterministic",   // "deterministic" | "model-judge" | "human"
    grader: "my-checker",
    scale: { label: "on-time" },   // optional display label
  }
);

Grades are often post-hoc, so this works even after the run has ended. With the artifact namespace off it is an inert no-op, never an error on your path.

From a non-SDK surface (HTTP)

When the judgment originates somewhere the SDK doesn't run — an end user in your web app, a review tool written in another language — post it to POST /v1/grade on modelrig-server:

curl -X POST https://api.modelrig.ai/v1/grade \
  -H "Authorization: Bearer $MODELRIG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"episode": "run-123", "score": 0.2, "comment": "CAGR off by 10x", "source": "app-user"}'

• Authenticated, never public. The request carries a rig_sk_ key holding the grade scope; there is no unauthenticated grade endpoint.
• The org rides the key, not the body. Which organization the row belongs to is resolved from the credential — you cannot file a grade into another org's corpus, and any org named in the body is ignored by construction.
• Same fields as the SDK call: {inferenceId | episode, score, scale?, comment?, kind?, source?}. A missing or out-of-range score, or a target naming neither an inference nor an episode, is a 422 — rejected before authorization runs.
• Response: { id, ts, org_id } — the stored row's id, its timestamp, and the org it landed in.

The ladder (rungs 0–1 today)

• Rung 0 — implicit, free with traffic: conformance / repair / refusal rates per route, computed from telemetry. Enough for schema-mechanical proposals: the bake-off gate (conformance CI + effective cost) does the judging.
• Rung 1 — explicit grades: a route with ≥10 attached grades. Routes with no schema to conform to (prose outputs) gain proposal confidence only this way, and every grade counts — human, model-judge, or deterministic.
• Rungs 2+ — judges: value accuracy beyond mechanical conformance; a later phase. The protocol shape above does not change — judges are just another model-judge grader.

Coverage renders per route on the console's /coverage page.

On the run page

Open a run in the console and its grades render as a chip: the deterministic run-outcome@v1 (its derived ✓/✗ plus the failure label) always, and the model judge's score and comment when one has been run. The same page lists the grades attached to the run — end-user thumbs joined by the run's episode, or by one of its steps' inferences when there is no episode — each shown as its score with the route-derived pass badge.

The improvement path

A grade is not only a gate — the comment is written guidance saved for a next run. A grade of 0.2 with "CAGR off by 10x" is the raw material for coaching the model that produced it: the score says how far off, the comment says how. Turning a route's low-scoring grades and their comments into the next attempt's context is the loop's natural next beat — the reason the comment field exists.

Privacy

Grade rows carry your score and comment — never the model's input or output text (those live only in the capture store, and only where a route opts in and your content custody posture allows it). The mirror is subject to the same export isolation as all telemetry.