Make a skill measurable

(needs modelrig 0.13.0 or newer) Your company's automations are built from Agent Skills — SKILL.md folders a coding agent loads. ModelRig gives each skill a receipt: what a run costs, how often its drafts pass, what they achieved, and how much review each of its tasks still needs — version by version. One block in the SKILL.md and one wrapper around its entry point are all it takes. ModelRig records and measures; it never runs, sends or publishes anything.

1. Set up

You need a ModelRig API key in the environment of the process that runs the skill:

Mint a key in the org you want the data in. Sign in at https://app.modelrig.ai, check the organization name in the header badge, and create a key on the API keys page. The rig_sk_ secret is shown once. One key = one org: every row the key exports lands in the organization it was minted in, whichever organization you happen to be viewing in the console. Set it on every service, worker and cron that creates a rig; your provider keys stay where they are.

# Wherever your app reads configuration: .env.local, your platform's env vars, a secret manager. Never in code.
MODELRIG_API_KEY=rig_sk_…

2. The frontmatter block

Agent Skills carry an arbitrary metadata map. ModelRig reads one block in it:

---
name: vendor-po-followup
description: Chase overdue purchase orders with vendors…
metadata:
  modelrig:
    skill: vendor-po-followup     # defaults to name
    version: "3"                  # free string; bump on behaviour change
    tasks: [po-followup-draft, po-reply-classify]
    review: earned                # optional; seeds the review gate on first sight (never overwrites)
---

• skill is the id receipts group by — an identifier (no spaces, ≤ 128 characters). It defaults to name.
• version is a short label (≤ 32 characters, no spaces). Bump it whenever the skill's behaviour changes; the receipt compares versions side by side.
• tasks are the steps or routes whose drafts are graded and reviewed — the names you pass to tags.step, rig.reviewDecision and rig.approval.
• review (earned or always) is optional. See section 5.

readSkillMetadata(skillMdText) parses it: a pure function over the frontmatter only (no filesystem, no network) that returns { name, description?, modelrig?, problems } — a SkillMetadata — or null when the text has no frontmatter with a name (describeFrontmatterProblem says why in one phrase — a duplicate key, invalid YAML, no name). A missing metadata or metadata.modelrig is fine (modelrig is then absent); a block that does not read is dropped whole and named in problems. findFrontmatter returns the frontmatter's span, and skillTagValue / parseSkillTag convert between { id, version } and the <id>@<version> tag value.

3. Wrap the entry point: rig.skill

await rig.skill({ id: "vendor-po-followup", version: "3" }, async () => {
  // everything the skill does
});

Inside fn:

• Every call — rig.run, rig.runRaw, rig.recordCall — carries the reserved tag skill=<id>@<version>, under your own tags (a per-call skill tag wins).
• Every run started inside — run.start, run.scope, rig.process, rig.conversation opening a new run — records RunMeta.skill = { id, version } unless it names its own (rig.process(id, { skill })).
• It nests either way round with rig.process and rig.conversation, and with itself (the innermost skill wins). Concurrent rig.skill scopes never cross-attribute. Outside a scope every row is byte-identical to before.

rig.skill returns whatever fn returns. A bad { id, version } throws a teaching error before fn runs. (runInSkill and currentSkill are the same mechanism as standalone exports.)

4. The receipt

Open See → Skills in the console, or read GET /v1/console/skills / GET /v1/read/skills (a rig_sk_ key) and …/skills/<skill> for one skill, with ?window=7d|30d|90d (default 30 days). Each receipt shows, for the skill and for each version side by side:

Rates and the cost spread are withheld below 20 runs, and the run count is shown instead. A skill whose calls all ran outside any run (0 runs) is held to the same 20 on each rate's own count — graded calls, or recorded outcomes — and the receipt says so (threshold_basis: "graded"). Reads are capped; a receipt that hit a cap says so and its numbers are floors. The tasks on a receipt are the routes and caller-chosen tags.step names its calls used, the tasks its approvals name, and any you add with ?task= (at most 10).

5. The review gate where the skill sends anything outward

If the skill drafts something that leaves the company — an email to a vendor, a reply to a customer, an order — put a review gate in front of it and record the employee's verdict. The snippet the scaffold tool returns (below) does exactly that:

await rig.skill({ id: "vendor-po-followup", version: "3" }, async () => {
  const segment = rig.process(processId, { name: "vendor-po-followup", party: "vendor" });
  const draft = await rig.runRaw({ provider, model, apiKey, systemPrompt, userPrompt, tags: { step: "po-followup-draft" } });
  const gate = await rig.reviewDecision({ task: "po-followup-draft", subjectKey: processId });
  if (gate.decision === "auto") await send(draft.output); else await queueForReview(draft, gate); // your code sends; ModelRig never does
  segment.wait({ party: "vendor", reason: "awaiting_reply" });
});
// later, when an employee decides in YOUR dashboard:
rig.approval({ task: "po-followup-draft", subject: { inference: draftInferenceId }, verdict: "approved", reviewer: employeeRef });

To start each task with a review-gate mode from the SKILL.md, seed it — once, explicitly:

const meta = readSkillMetadata(readFileSync("SKILL.md", "utf8"));
if (meta) await rig.seedSkillReviewGates(meta);

rig.seedSkillReviewGates sets review_gate.mode from metadata.modelrig.review only for a task that has no policy yet — it never overwrites one an employee or the console set. It needs a key with the task-settings scope; without it (or before the server can store review gates, or for a task ModelRig has not seen yet) each task is skipped. It never throws; it returns a SeedSkillReviewGatesResult — { seeded, skipped: [{ task, reason }] }. It is never automatic (makeSeedSkillReviewGates builds the same function for a custom client). A task whose route YAML declares grade: { review_floor: high_risk } is never seeded (the server refuses with 409; it is skipped as code_floor, and every draft stays reviewed). A running process sees a newly seeded policy once its 60-second reviewDecision cache expires.

Two starter guides fit outward drafts: vendor follow-up and customer escalation.

6. Let your coding agent do it: scaffold_skill_receipts

The modelrig-oracle MCP server has a tool for an agent authoring a skill. It needs the matching modelrig release (they ship together, the SDK first); with an older modelrig installed it answers a teaching error naming the upgrade. scaffold_skill_receipts({ skillMd, tasks? }) returns draft files and notes — it writes nothing; the agent writes them:

1. SKILL.md with the metadata.modelrig block inserted and every other byte preserved (unchanged if a valid block is already there); 2. modelrig/grading-guides/<task>.json — a starter grading guide per task from the closest use-case template by keyword overlap with the skill's name and description, or a blank guide shape; 3. modelrig/golden-checklist.md; 4. modelrig/instrumentation.md — the snippet above, with the skill's names.

Then apply each guide with set_task_settings, wrap the entry point in rig.skill, and the receipt appears on the first day the skill runs.