Artifact instrumentation (runs, steps, artifacts)

ModelRig can record the execution graph of any pipeline your code runs — the run, its steps, and the artifacts each step produced (the prompt, the raw model text, the parsed output, the evidence it grounded on) — with lineage, grades, and an integrity hash for each. The console then renders that run as a semantic chain you click through: what your pipeline actually did.

Recording your workstream is the recommended, normal path — it is what the run → step → artifact graph and every downstream analysis are built on — so it is on by default when a control plane is configured (an api-mode rig_sk_ key, or the direct-mode Supabase pair) — metadata + hashes only. A pure local-only rig (no sink to ship to) stays off. Either way it is additive, and the opt-out is one account setting: set posture off in account settings (the console or the set_org_settings MCP tool) — the account posture is the customer control, not a deployment env var. (A pure local-only rig with no control plane is inert either way — run.start a no-op, artifact.save null, no rows written, your pipeline byte-for-byte as before.) modelrig status prints the current posture (on/off + why). Storing the artifact content bytes on top of this metadata is the second axis — the recommended, consent-gated content custody path — never blurred with the telemetry axis here.

Metadata + hashes only, this release. The serialized value you save is hashed (sha256) and the bytes are discarded — nothing but the metadata row is persisted. Content custody (the blob path) ships in a later release; the hash proves integrity today. A zeroRetention run refuses every artifact fail-closed at the SDK gate.

The namespace toggle

# On by default once a control plane is configured — nothing to set. The CUSTOMER
# opt-out is the account posture (set posture off in account settings / the
# set_org_settings MCP tool), not this env var.
export MODELRIG_ARTIFACTS=1                  # local-only namespace toggle: force ON
                                             #   with =1 on a rig with no control plane
export MODELRIG_ARTIFACT_STEPS=step_a,step_b   # optional: instrument a subset
                                            # (unset/empty = every instrumented step)

Rows land in the local telemetry buffer and export to the control plane through the standard sink (MODELRIG_API_KEY + MODELRIG_INGEST_URL, or the direct-mode Supabase pair). With no sink configured they stay local.

An agent can do everything here except mint that MODELRIG_API_KEY — creating it requires a human console login at <https://app.modelrig.ai>, so an agent-driven setup should plan that one handoff rather than stall on it.

Instrument a pipeline

The namespace hangs off rig.artifacts. Two moving parts: a run context around the whole pipeline, and a per-step save of the work products.

import { createRig, loadConfigFromEnv } from "modelrig";

const rig = createRig(loadConfigFromEnv());

// 1. Open a run context around the pipeline. episodeKey bridges grades;
//    it is a grouping key, not a unique id — a natural choice is your own
//    per-run attempt id.
const run = rig.artifacts.run.start({
  pipeline: "my-pipeline",
  episodeKey: runAttemptId,
  environment: process.env.NODE_ENV,
});
try {
  for (const step of steps) {
    const prompt = assemble(step);
    const result = await callModel(prompt);          // your own model call

    // 2. Save the step's artifacts. stepKey groups them in the console's
    //    chain view; task carries the semantic task name.
    const p = rig.artifacts.artifact.save(prompt, { name: `${step}.prompt`, type: "prompt", stepKey: step });
    const raw = rig.artifacts.artifact.save(result.rawText, { name: `${step}.raw`, type: "raw_response", stepKey: step });
    const parsed = rig.artifacts.artifact.save(result.json, {
      name: `${step}.output`, type: "step_output", task: step, stepKey: step, schema: step.schema,
    });

    // 3. Lineage: the parsed output derives from the prompt + raw; this step's
    //    prompt consumed the previous step's output.
    if (p && parsed) rig.artifacts.artifact.link(parsed, p, "derived_from");
    if (raw && parsed) rig.artifacts.artifact.link(parsed, raw, "derived_from");

    // 4. Grades (optional, deterministic): flag conformance, degraded
    //    provenance, anything you can decide without an LLM judge.
    if (parsed) rig.grade({ kind: "artifact", id: parsed.id }, {
      score: conforms(result.json) ? 1 : 0, kind: "deterministic", grader: "schema-check",
    });
  }
  run.end("succeeded");
} catch (err) {
  run.end("failed");
  throw err;
}

For concurrent runs in one process, use rig.artifacts.run.scope(opts, fn) instead of start/end — each scope keeps its own ambient context so overlapping runs never cross-attribute. run.start uses enterWith, which is right for one run at a time on a request/async path.

Steps vs. artifacts

• An artifact is a work product — a durable thing your pipeline made. It always saves (metadata + hash), whether or not the model call went through the rig.
• A step is an execution row. rig.run() auto-creates a ground-truth step (its inference is real, so cost and latency are measured) when it runs inside a run context. A stage whose model call does not go through the rig produces artifacts with step_key set and no step row — which is correct and honest. Never route a stage through the rig just to mint a step row; the console renders "cost n/a" for un-routed steps rather than inventing a number.

Tags, dimensions, and the graph

ModelRig has two grouping mechanisms, and they compose:

• Tags are a flat filter/rollup. Every rig.run / rig.runRaw takes a tags object; those key/values ride the telemetry row and drive /costs and the route dashboard (group cost by project, by environment, by whatever you tag).
• The graph is the per-run tree at /runs/[id] — the ground-truth steps a run recorded, in order, with their inferences, costs, and the artifacts attached to each step.

Declare your dimensions in modelrig/rig.yaml so the cost views know which tags matter:

name: my-rig
dimensions:
  - { key: project,     label: Project,     required: true }
  - { key: environment, label: Environment, required: false }

A required dimension that is missing from a run's tags is warned once per process (required dimension "project" missing from run tags — costs … will be under-attributed on /costs) — on the routed lane and the raw lane alike. It is a hygiene warning, never an error: the run proceeds.

Conventional tag keys (documented, not enforced): subject (your end-customer — the tenant key), feature, project (flat grouping), step (names the step this call belongs to), and run_id (bridges a call to a run's episodeKey). subject is your end-customer's opaque id — it powers the per-customer tenant statements on /tenants; a tag named client/tenant is only a cost dimension. (Hash it yourself if it looks like PII — see tenants.md.) Inside a run.start / run.scope context the SDK stamps both run_id and step for you — run_id = the run's episodeKey, or its id when you gave none; step = the current step key (your explicit stepKey, else tags.step, else the auto route#n / provider/model#n key) — on every attempt row of both lanes, unless you passed your own value (yours always wins). Outside a run context nothing is stamped, so the row is byte-identical. This is why runs are the standard path: a run.start alone fills both /runs (the graph) and /projects (the tag tree), with no hand-written id. ModelRig owns lane, zdr_enforced, rung, and — on the hosted gateway — key_ownership; you cannot override those.

How they compose. Inside a run.start context, both rig.run and rig.runRaw record a ground-truth step and carry the run's run_id tag, so the same calls appear on /runs/[id] (the graph) and under that run on /projects (the tag tree). Name it with stepKey (either lane) — explicit stepKey wins, else tags.step, else an auto key (route#n / provider/model#n). An artifact.save without a stepKey lands in the ambient step (the most recent step in the context), so a raw call and the artifacts you save around it end up in one group on /runs/[id]:

const run = rig.artifacts.run.start({ pipeline: "report" });
// A raw (BYOK) call names its step; the save with no stepKey joins it.
const out = await rig.runRaw({ provider, model, apiKey, systemPrompt, userPrompt,
  tags: { project: "acme" }, stepKey: "earnings_5a" });
rig.artifacts.artifact.save(out.output, { name: "earnings.output", type: "step_output" });
run.end("succeeded");

Raw lane and the tree. A rig.runRaw / rig.runRawStream call inside a run context records a ground-truth step since modelrig 0.5.0, on success and on failure. Before 0.5.0 the raw lane never read the run context, so a pipeline that wrapped raw calls in run.start got an empty tree. (Raw steps are metadata only — no artifact content is written on the raw lane.)

Live call-config and A/B arms

Instrumentation records what your pipeline did; call-config changes which model does it, live, without a redeploy — and the two meet on the telemetry row. When a route runs a live A/B (mode='experiment'), the arm that served each call is recorded on inferences.meta.arm, so the run → step → artifact graph and the console's A/B readout can group outcomes by arm. This section is how that field gets populated; the call-config API itself (the resolver, the fail-open contract, the console card, and the MCP write path) is in the quickstart's Change your route's model live.

Overlay live call-config onto a bundle. A route's live config is read just-before-call through a resolver and merged over the bundle you serve through rig.runBundle() (never rig.runRaw — the raw lane owns no bundle to overlay). The read is fail-open and, with no override, byte-identical to the deploy-time bundle:

import {
  createApiCallConfigFetcher, createCallConfigResolver, resolveCallConfigForRun,
} from "modelrig";

const resolver = createCallConfigResolver({
  fetcher: createApiCallConfigFetcher({ ingestUrl, apiKey: process.env.MODELRIG_API_KEY! }),
  ttlMs: 30_000,   // cached; ETag-revalidated — no per-call network cost when fresh
});

// Per call: resolve the config, draw an arm locally (pin/no-override ⇒ arm is null).
const resolved = await resolver.getCallConfig(route);
const { bundle: runnable, arm } = resolveCallConfigForRun(bundle, resolved, { input });

// Thread the arm onto the run so it lands on inferences.meta.arm.
await rig.artifacts.run.scope({ pipeline: "report", episodeKey: runId }, async () => {
  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, arm, stepKey });
  rig.artifacts.artifact.save(out.output, { name: "section.output", type: "step_output" });
});

How meta.arm gets populated. resolveCallConfigForRun(bundle, resolved, { input, unitKey }) returns { bundle, arm }: for an experiment it draws one weighted arm and returns its stable name; for a pin, a null/live/malformed resolution, or no override it returns the bundle unchanged and arm: null. You pass that arm on RunOptions.arm, and the attempt-meta producer stamps it onto every recorded row's meta.arm for that call (a non-meta-safe value is dropped). Absent arm ⇒ no meta.arm — byte-identical to a non-experiment run. Drawing is sticky: pass unitKey (or let stickyKey read a field from input) and the same unit always draws the same arm, so the A/B is measured honestly across calls.

The console reads per-arm outcomes at GET /v1/console/ab-readout?route=<route> (samples, conformance, and cost grouped by arm; non-experiment rows, which carry no arm, are ignored). meta.arm sits in the recorded-metadata inventory alongside the rest of the attempt envelope — see content custody.

Never throws into your pipeline

Every entry point is fail-open: a telemetry-buffer write that fails is logged and dropped, never thrown into your code. Instrumentation that runs after an expensive model call must never be the thing that kills the run.

See it

The console's Runs tab lists every instrumented run; /runs/[id] is the semantic chain (artifacts grouped by step, with hashes, costs, and grade badges) and /artifacts/[id] shows one artifact's lineage, versions, and grades. Enabling the flag also lights up the /setup artifacts line.

Grades (rig.grade)

Capture records what ran; a grade records whether it was good. A grade is a 0–1 score (with an optional comment) your code — or a human, or an AI — attaches to a subject: one model call (an inference), a whole run (an episode), or an artifact. It is the signal per-route optimization coverage is built from — the evidence level that decides how much confidence a swap proposal carries.

// One call — the inference id rides on every RunResult:
const result = await rig.run("example.support_summarize", { input, tags: { run_id } });
rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" });

// A whole run (the subject id is your run_id tag):
rig.grade({ kind: "run", id: run_id }, { score: 0.2, comment: "CAGR off by 10x", kind: "human" });

• The subject is { kind: "inference", id } (result.meta.inferenceId), { kind: "run", id } (the run's run_id/episodeKey tag), or { kind: "artifact", id }. opts is `{ score, scale?, comment?, kind, source? } — the score in [0, 1]` is the primitive (up = 1, down = 0), and the pass/fail verdict is derived from it against the route threshold. A subject naming no known inference, run, or artifact attaches to nothing — attribution is never guessed.
• Emit it from a signal you already have. Most pipelines already decide pass/fail somewhere — a schema validator, a downstream accept/reject, a human QA gate. Emit rig.grade from that (score 1 on a clean pass, 0 on a rejection) rather than adding a new review step.
• Never on your hot path. The write is synchronous and local (SQLite), never a network call and never a failure your code can see; the cloud mirror rides the telemetry exporter's normal cadence.
• Why it matters. A route with ≥10 attached grades reaches Rung 1 on /coverage; a route with no schema to conform to (a prose output) gains proposal confidence only this way. Grades are also what let /insights surface a task that is degrading.
• From a non-SDK surface (an end user clicking thumbs-down in your web app), post the same fields to POST /v1/grade with a rig_sk_ key carrying the grade scope — the org is resolved from the key, never the body. See the grade protocol for the full row shape and the ladder.

Promote a golden, then bake it off (promoteToEval → --from-eval-cases)

A run's artifacts are the raw material for a durable eval case. Promote a real output into the org's eval suite and it becomes replayable evidence — the path a raw/run-based pipeline (a rig.runRaw seam with no declared route) uses to prove a cheaper model, alongside the route-scoped bakeoff --route … --replay-last N.

// Promote an artifact you saved this run into the org's eval suite.
const caseId = rig.artifacts.artifact.promoteToEval(handle, { task: "report.section" });
// `corrected` set ⇒ a golden-corrected pair; absent ⇒ a failure case.
// Unresolvable task ⇒ a warned no-op (returns null, never throws).

# Replay the promoted eval cases through variants — the raw/run-based lane.
modelrig bakeoff --from-eval-cases report.section --variants default,cheap

• promoteToEval(ref, { task?, corrected?, origin? }) binds the case to a route by task (from opts, else the task the artifact was saved with this session), and is also reachable from the console's one-click "make golden".
• --route needs a declared route; --from-eval-cases <task> is the path for a runRaw seam that owns no route. Either way a bake-off measures — it never switches what serves a route; a swap is a human-approved PR.

A reusable per-step seam

Most pipelines want one small helper that saves the three work products of a step and wires their lineage, so the call sites stay a single line. modelrig observe [path] --scope <one pipeline> generates this seam for you (a record-step file) plus an OBSERVE.md whose per-call-site wiring is emitted as diffs to apply — it never edits your source. The seam is equally yours to write by hand; here it is, self-contained:

import { createRig, loadConfigFromEnv, type Rig } from "modelrig";

const rig: Rig = createRig(loadConfigFromEnv());

/** Save prompt + raw + parsed for one step, link lineage, flag conformance. */
function recordStep(
  stepKey: string,
  work: { prompt: string; rawText: string; json: unknown; conforms: boolean },
): void {
  const p = rig.artifacts.artifact.save(work.prompt, { name: `${stepKey}.prompt`, type: "prompt", stepKey });
  const raw = rig.artifacts.artifact.save(work.rawText, { name: `${stepKey}.raw`, type: "raw_response", stepKey });
  const parsed = rig.artifacts.artifact.save(work.json, { name: `${stepKey}.output`, type: "step_output", task: stepKey, stepKey });
  if (p && parsed) rig.artifacts.artifact.link(parsed, p, "derived_from");
  if (raw && parsed) rig.artifacts.artifact.link(parsed, raw, "derived_from");
  if (parsed) rig.grade({ kind: "artifact", id: parsed.id }, { score: work.conforms ? 1 : 0, kind: "deterministic", grader: "schema-check" });
}

Call recordStep(step, { … }) once per stage inside the run.start / run.end context above, and every entry point stays fail-open — a save that fails is logged and dropped, never thrown into your pipeline.

In-repo example (reconstructed via onboarding-test run-2). ModelRig's own customer-zero pipeline (the InferWealth v3 report generator) was stripped back to a clean slate to be re-onboarded through this exact doc. Its live seam is reconstructed as part of onboarding-test run-2; until then, the snippet above is the reference. Do not resurrect a deleted seam from git history — writing a fresh one is the point.