Review gates that earn autonomy

(needs modelrig 0.13.0 or newer) Your automations draft emails, quotes and purchase-order replies that an employee checks before they go out. A review gate answers one question for your own code, per draft: does this still need a human? — review, sample or auto, with a receipt that says why. Every approval your employees record teaches the task's grader; once that grader is calibrated against them, an earned gate keeps only a deterministic sample in front of a person.

ModelRig never sends, holds or approves anything. The gate only answers; your code decides what review, sample and auto mean.

1. Set up

You need a ModelRig API key in the environment of the process that asks the gate and records approvals:

Mint a key in the org you want the data in. Sign in at https://app.modelrig.ai, check the organization name in the header badge, and create a key on the API keys page. The rig_sk_ secret is shown once. One key = one org: every row the key exports lands in the organization it was minted in, whichever organization you happen to be viewing in the console. Set it on every service, worker and cron that creates a rig; your provider keys stay where they are.

# Wherever your app reads configuration: .env.local, your platform's env vars, a secret manager. Never in code.
MODELRIG_API_KEY=rig_sk_…

The task is the step or route whose drafts are reviewed (for example po-followup-draft). It needs a grading guide before approvals can calibrate its grader — see Set up a grader.

2. The decision table

The policy lives on the task (review_gate; off until you set it):

{ "mode": "earned", "sample_rate": 0.1, "floor": "none" }

sample_rate is between 0.02 and 1. The sample is a hash of your own key for the draft, so the same draft always gets the same answer. auto is only ever possible at CALIBRATED or JUDGE_LED, with mode: earned and no floor.

Every answer is a ReviewDecision — the receipt:

{
  decision: "review" | "sample" | "auto",
  reason: { code: "calibrated_sampled", message: "The grader is calibrated and this draft falls in …" },
  ladder_state: "CALIBRATED", agreement: 0.92, ci: [0.84, 0.96], n: 61, policy_version: 1,
}

3. Ask the gate

const decision = await rig.reviewDecision({ task: "po-followup-draft", subjectKey: po.id });

if (decision.decision === "review" || decision.decision === "sample") {
  await queueForApproval(draft, decision); // your own dashboard
} else {
  await sendDraft(draft); // your code acts; ModelRig never does
}

• subjectKey is your own id for the draft (the PO number, the ticket id). It is hashed in your process and never leaves it: rig.reviewDecision reads only the task's policy and grader standing from GET /v1/read/review-decision?task=, then computes the sample bit locally.
• The read is cached for 60 seconds per task and times out after 300 ms.
• The server also caches each task's calibration for 30 seconds, so a grader demotion (for example to DRIFT_CHECK) reaches running processes within about 90 seconds in the worst case. The deterministic sample keeps a share of drafts reviewed throughout.
• Fail-closed. No API key, a timeout, an HTTP or network error, an unreadable answer, no policy — every one of these answers review. It never throws.
• A floor in code. rig.reviewDecision({ task, subjectKey, floor: "high_risk" }) answers review immediately, with no network call. A route can also declare one in its YAML, which the console cannot lower. The YAML floor holds in the process that loaded the route immediately, before any network call — whatever the server has synced (another deployment of the same route without the floor cannot lift it):

grade:
  pass_threshold: 0.7
  review_floor: high_risk

4. Record approvals from your own dashboard

When an employee approves, edits or rejects a draft in your dashboard, record it:

rig.approval({
  task: "po-followup-draft",
  subject: { artifact: draftArtifact.id }, // or { inference: result.inferenceId }
  verdict: "edited",                       // "approved" | "edited" | "rejected"
  edited: sentText,                        // only with verdict "edited"
  reviewer: employee.id,                   // an opaque id from your system
});

• An approval is a human grade on the draft: approved scores 1; edited and rejected score 0 (an edited draft was not good enough as drafted). It is stored with source: approval:<task> and grader: approver:<ref>, and counts wherever a human grade counts — the grader's labels, its agreement, the grade queue (Grade → Approvals lists them).
• edited also keeps the edit's value: the edited text is saved as an artifact (through content custody and its scrub) and becomes a **human-corrected eval case** on the draft artifact. Pass the draft as { artifact } for this; with { inference } the grade is recorded and the result's evalCaseSkipped says why no golden was kept.
• rig.approval refuses any subject other than an inference or an artifact, with a teaching error.
• **reviewer is pseudonymous — pass an opaque employee id, never a name or email; a hash of a known email can be matched.** As a safety net, a reviewer that contains @ or whitespace, is longer than 48 characters, or uses characters outside letters, digits, ., _, : and - is replaced by the first 12 hex characters of its sha256 before it is stored — the raw string is not kept, but that is pseudonymisation, not anonymisation. A value like john.smith is stored as given.

The ModelRig grade queue is optional — approvals from your own UI are enough.

5. Turn on auto-grading for the task so drift is caught

Agreement needs judge grades on the same drafts your employees approved. Turn on auto-grading for the task (the Calibration tab of its grader, or set_task_settings with grade_auto), so drafts sent as sample or auto keep being graded by the judge. If the judge drifts from your employees, the grader drops to DRIFT_CHECK and the gate demotes itself to review on its own.

6. Set the policy

• Console: the grader's page (Graders → your task → Calibration) has a Review gate card — the current decision and why, how many approvals the task still needs to graduate (from the agreement target), the decisions the gate would have made over the last 30 days (counts, never actions), and the policy editor.
• API: PATCH /v1/tasks/:task/settings with { "review_gate": { … } } (or null to turn it off) — the same auth as grade_auto: an admin/owner session, or a rig_sk_ key with the task-settings scope. A policy that would sit below a route's review_floor is refused with 409.
• MCP: get_review_gate and set_review_gate (modelrig-oracle).
• In code, the pure pieces are exported too: decideReview, parseReviewGate, validateReviewGate, reviewGatePosture, and the ReviewGatePolicy and ApprovalInput types; approverRef shows exactly what a reviewer id is stored as.

7. Deploy order

Apply control-plane migration 00090 (tasks.review_gate) → deploy the server → upgrade the SDK. Until the server sees the column (it re-checks about once a minute; no restart), a review_gate PATCH answers 503 with a teaching message and every decision is review.

8. What the gate never does

• It never sends, holds, approves or blocks a draft — it answers.
• It never calibrates per reviewer: one grader per task.
• It never changes its own policy: a person sets it; the gate only reports.