Conversations, outcomes and limits

(needs modelrig 0.12.0 or newer) Three additions for chatbots and agent loops, all on the same loop you already use — see, grade, improve:

1. Conversations — wrap a chat (or an agent's task loop) in one call and every turn lands in ONE run you can read and grade as a whole. 2. Outcomes — post what actually happened later (resolved, booked, approved, CTR) from a webhook; it joins its conversation whenever it arrives. 3. Run limits — stop a runaway loop in-process before the bill does.

None of this is a trace-tree debugger: a conversation is shown as a transcript (turns in order), not a span waterfall.

1. Set up

You need a ModelRig API key in the environment of the process that runs your agent:

Mint a key in the org you want the data in. Sign in at https://app.modelrig.ai, check the organization name in the header badge, and create a key on the API keys page. The rig_sk_ secret is shown once. One key = one org: every row the key exports lands in the organization it was minted in, whichever organization you happen to be viewing in the console. Set it on every service, worker and cron that creates a rig; your provider keys stay where they are.


# Wherever your app reads configuration: .env.local, your platform's env vars, a secret manager. Never in code.
MODELRIG_API_KEY=rig_sk_…

If your code calls providers directly (no route files), build a raw-lane rig:

No routes yet? Build the rig raw-only. Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with routesDir: null: rig.runRaw, rig.runRawStream and rig.recordCall work at once, and rig.run / rig.runBundle refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.


import { createRig, loadConfigFromEnv } from "modelrig";

// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });

2. Wrap a chatbot conversation


// One conversation = one run (pipeline = agent, episode_key = your conversation id).
const convo = rig.conversation(conversationId, { agent: "support-bot", user: userId, tenant: accountId });

await convo.turn(async () => {
  // every rig.run / rig.runRaw / rig.recordCall in here records as step turn:<n>.<k>
  const reply = await rig.runRaw({ provider: "openai", model: "gpt-4o-mini", apiKey, systemPrompt, userPrompt });
  return reply;
});

// When the conversation is over (optional — idle conversations close themselves):
convo.end();

The turn rule. turn(fn) opens turn n + 1; each call inside it records under step key turn:<n>.<k> (k = the call's index within the turn — a repeated step key would read as a retry). A call made outside any turn() attaches to the current turn (turn 1 if none was opened). An explicit stepKey (or tags.step) always wins. user and tenant become the user and subject tags on every call.

Re-entry. A later request that calls rig.conversation with the same id and agent continues the SAME run (same process, or the same local store after a restart), and turn numbering continues. Two workers with separate stores can produce two runs for one conversation; the console groups them by conversation id.

Closing. Conversations rarely end cleanly, so the server closes a running conversation once its last call is older than the org's idle window (default 30 minutes) and marks it succeeded. A conversation is then graded once, whole, under the task conversation:<agent> — set a grading guide on that task (the support-conversation template is a starter). Conversations that arrive through OTLP (/v1/otlp/v1/traces) appear on the Conversations page but are not yet offered to the conversation auto-grader; grade them from the console for now.

3. Post an outcome from a webhook

An outcome is a grade with kind: "outcome": a name (resolved, meeting_booked, ctr.7d), a 0–1 score (booleans are 1/0), and optionally the raw business value.


curl -X POST https://api.modelrig.ai/v1/outcomes \
  -H "authorization: Bearer $MODELRIG_API_KEY" \
  -H "content-type: application/json" \
  -d '{"key":{"conversation":"chat-42"},"name":"resolved","score":true,"idempotency_key":"evt_123"}'
# → 202 {"id":"…","subject":{"kind":"run","id":"chat-42"}}


rig.outcome({ key: { conversation: "chat-42" }, name: "resolved", score: true, idempotencyKey: "evt_123" });
// a continuous number: the score is derived as value / scale.max, clamped to 0–1
rig.outcome({ key: { inference: inferenceId }, name: "ctr", value: 0.034, scale: { max: 0.05 } });

key is exactly one of { conversation }, { run }, { inference }, { artifact }. A conversation key is stored against the conversation id, so the outcome can be posted before, during or after the conversation's telemetry arrives — it joins at read time; nothing is pending or dropped.
idempotency_key makes a retried webhook safe: the same key always maps to the same outcome, never a duplicate.
The key needs the ingest scope. The MCP oracle's record_outcome tool does the same.
idempotency_key is scoped by outcome name: one webhook event can post resolved and csat under the same key and both land; the SDK and the HTTP API produce the same id for the same (name, key), so a retry through either door never duplicates. With an idempotency key, the id that rig.outcome returns is a local id: the SDK does not know your org, so the server stores the row under the org-scoped id (the one POST /v1/outcomes returns). Look outcomes up by their subject and name, not by the SDK-returned id.
If your organization's capture posture is off, outcomes are refused (403) like every other telemetry.

Deploy order (self-hosting). Outcomes need control-plane migration 00089. The order is fixed: apply 00089 → deploy the server → then upgrade the SDK that calls rig.outcome. Until the server sees the migration (it re-checks about once a minute, no restart needed), POST /v1/outcomes answers 503 so your webhook retries, and outcome rows an SDK exports are dropped and counted (outcome_schema_unavailable in the ingest response) while the rest of its telemetry keeps flowing.

Outcome agreement. On a grader's page, the console shows how often the grader's pass/fail matches what actually happened — held-out, with n and a 95% interval, and "not enough data" below 20 pairs. It is a receipt: an outcome never moves a grader to calibrated on its own (only your own grades do).

4. Set run limits


const run = rig.artifacts.run.start({
  pipeline: "coding-agent",
  limits: { maxUsd: 2, maxCalls: 60, maxWallMs: 10 * 60_000, maxRepeatToolCall: 4 },
});
// …or on a conversation:
rig.conversation(id, { agent: "support-bot", limits: { maxUsd: 0.5 } });

Before each rig.run / rig.runRaw, the SDK checks the run's accumulated cost, call count and elapsed time, and whether the model has asked for the same tool with the same arguments maxRepeatToolCall times in a row (only a hash of the arguments is kept). On a breach the call is not sent: it throws RunLimitExceeded (failure class limit_exceeded), the run ends failed with failure_reason limit:max_usd / limit:max_calls / limit:max_wall_ms / limit:max_repeat_tool_call, and later calls in that run keep refusing. rig.recordCall cannot stop a call your code already made — it counts toward the limits and the NEXT dispatch throws. maxUsd counts each call's model cost; it does not yet include web-search surcharge rows, so a grounded loop can exceed maxUsd by its search fees.

Limits are per run and immediate; envelopes are org/route money budgets enforced server-side. Use both.

In the console, Runs → stopped at limit and Conversations → Stopped at limit list the runs that hit a limit.