ModelRig Docs Leaderboard Console ↗ GitHub Sign up

Set up a grader

A grader is an AI judge for one step or route in your product. You teach it with a grading guide — 3–7 pass/fail criteria, one plain sentence each — and with your own grades. Once it agrees with you often enough, it can grade new outputs for you.

What you can grade

A grader belongs to a task: a named route or a named step.

GET /v1/tasks lists every task in your organization. Registration never overwrites an existing task, so repeating it is safe.

A step with the same name as a route shares that route's grader, including its auto-grade setting.

Let your coding agent do it

Copy these instructions into your coding agent (Claude Code, Cursor) to create a grader. It reads your key from MODELRIG_API_KEY (or your project's .env files) and never creates, rotates or prints keys itself. It shows you draft criteria before it writes anything, and edits an existing guide rather than replacing it. The same text is on the console's Graders page.


Create a ModelRig grader for one step or route in this project.

Rules: never create, rotate or request API keys yourself, and never print one.
Read a Standard rig_sk_ key from the MODELRIG_API_KEY environment variable or, if
it is not set, from this project's env files (.env, .env.local); if there is none,
stop and ask me to mint one at app.modelrig.ai -> API keys. HTTP base:
https://api.modelrig.ai, header "Authorization: Bearer $MODELRIG_API_KEY". If the
ModelRig MCP server (modelrig-oracle 0.3.0 or later) is connected,
prefer its tools; otherwise use the HTTP calls.

1. Pick the task. GET /v1/tasks lists my routes and steps (a step is the tags.step
   my rig.runRaw / rig.recordCall calls set) and the org_id the key belongs to. If
   none of this project's steps or routes are listed, stop and tell me: the key
   may be for another org. Ask me which task if it is unclear. If the one I name
   is not listed, find its tags.step in the code: it may be renamed or retired.
   Create a task (POST /v1/tasks {"task":"<TASK>"}; letters, digits, . _ : - only)
   only when the code sets that exact name and it has not run yet: a misspelled
   task gets a grader that never sees a call. Then read its settings: MCP
   get_task_settings {"task":"<TASK>"}, or GET /v1/tasks/<TASK>/settings.
2. If grading_guide is already set, show it to me and propose changes to it, not
   a new guide. A write replaces the whole guide: carry over every criterion you
   keep with the same id (my grades are matched to criteria by id), and keep
   grading_guide.teach_grade_ids.
3. Learn what good looks like: read the task's prompt, output schema and calling
   code in this repo, and note what the code already checks (validators, schema
   and length checks); criteria should cover what code cannot. List recent calls
   with GET /v1/read/inferences?route=<TASK> for a route, or
   GET /v1/read/inferences?tag=step&value=<TASK> for a step. Those rows are
   metadata (failures, retries, cost), not prompts or outputs: read 3-5 actual
   outputs from wherever this project stores them (database, logs, fixtures).
4. Draft 3-7 pass/fail criteria as {id, title, pass, fail}, one sentence each.
   Suggest "fatal": true for any criterion whose failure alone should fail the
   whole output. Show them to me. Do not write anything until I approve.
5. Write the guide: MCP set_task_settings {"task":"<TASK>","grading_guide":{...}},
   or PATCH /v1/tasks/<TASK>/settings with
   {"grading_guide":{"criteria":[...],"teach_grade_ids":[...]}}.
   On a 422, fix the shape from fix_hint. Never send pass_threshold.
6. Verify: read the settings again (grading_guide.version went up), then
   GET /v1/read/calibration?route=<TASK> (steps use route= too; MCP resource
   modelrig://calibration/<TASK>) shows calibration.ladder_state TEACHING.
7. Tell me the next steps: grade about 50 outputs in the console's Grade queue, then
   press Re-measure on Graders -> <TASK>; turn auto-grade on once it shows
   CALIBRATED (agreement at or above its target, 0.85 by default). If it stays
   below target, review the disagreements there, tighten the criteria they point
   to (back to step 2), and re-measure.
   Re-measure and auto-grade use metered AI calls in my org.
The MCP steps need modelrig-oracle 0.3.0 or later; without it the agent uses the HTTP calls. Running it uses your coding agent; ModelRig's grading cost starts only when you press Re-measure or turn on auto-grade.

The MCP route (get_task_settings / set_task_settings with a grading_guide) needs modelrig-oracle 0.3.0 or later; the earlier 0.2.0 refuses a guide-only write. Listing and creating tasks are HTTP calls today — there is no MCP tool for them yet.

Or do it in the console

1. Open Graders → New grader and pick the step or route. 2. Write the criteria in the Grading guide tab and save. The grader is now TEACHING. 3. Grade outputs in the Grade queue. Its teaching sample shows recent outputs from steps and routes that have a guide.

From teaching to calibrated

Re-measure and auto-grade use metered AI calls in your organization. Writing a guide and grading by hand cost nothing.

Which grades count