Template: support conversation

(needs modelrig 0.12.0 or newer) Grade a whole support conversation the way a QA lead reviews a ticket: was the customer's problem really solved, inside your policies, with the right tools and tone, and handed to a human when it should have been. This page is one of the use-case templates — a starting point, not a finished grader. Unlike the single-call templates, it grades a conversation — one run opened with rig.conversation(id, { agent }) and graded once, whole, under the task conversation:<agent> (see Conversations, outcomes and limits). The machine-readable source is node_modules/modelrig/templates/support-conversation.json.

Does this task fit?

Support conversations fit the shipped teach → calibrate → grade → optimize loop when four conditions hold. (1) Each conversation is wrapped in rig.conversation(id, { agent }), so its turns land as one run and the conversation is graded once, whole, when it closes — not turn by turn. (2) Your support lead can write down what a resolved conversation means for your product (not just one that went quiet), which is the grading guide below. (3) Someone who knows the queue can grade a batch of past conversations so the grader for the conversation:<agent> task can be calibrated against them; outcomes you post later (resolved, reopened) show beside it as a receipt, never as the calibration itself. (4) Content capture is on for the agent, so the grader and your reviewers can read the transcript; without it only metadata is graded. This template is a starting point for each of those, not a claim that support is solved — fill in your policies and your actions before you rely on it.

Starter schema

Replace every TODO with your own taxonomy; the enum slots ship without values on purpose, so an unfilled template refuses to look finished.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$comment": "STARTER SCHEMA for ONE assistant turn in the conversation. Replace every TODO with your own actions before use. The template runs as-is: free-text fields validate immediately. TODO markers show where adding your own enum values tightens grading to your playbook; do not copy example words from the docs.",
  "title": "SupportTurn",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "reply",
    "action",
    "escalate"
  ],
  "properties": {
    "reply": {
      "type": "string",
      "description": "What the assistant says to the customer on this turn."
    },
    "action": {
      "type": "string",
      "description": "The action the assistant takes alongside the reply (answer only, look something up, change something, hand off). TODO(enum): constrain this to YOUR own action names by adding an \"enum\": [ ... ] here. It runs as free text immediately.",
      "$comment": "TODO(enum): add your action names as the enum for this field."
    },
    "escalate": {
      "type": "boolean",
      "description": "True when this turn hands the conversation to a human. Pairs with the escalated-when-needed criterion in the grading guide."
    },
    "policy_refs": {
      "type": "array",
      "items": {
        "type": "string"
      },
      "description": "Optional: the ids of your own policy articles the reply relies on, so the grader can check policy adherence against them."
    }
  }
}

Starter grading guide

The starter criteria run as written — sharpen each one in your team's own words as you grade. They are in the exact jsonb shape the set_task_settings write path validates — binary checks, each with a one-sentence pass and fail. There is no pass_threshold here: that lives in the route's grade: block.

resolved_for_real — The problem was actually resolved

• Pass: the customer's stated problem is solved by the end of the conversation, by your own definition of solved — not merely that the customer stopped replying.
• Fail: The conversation ends with the problem unsolved, deflected, or only acknowledged, or it reads as closed only because the customer went quiet.

policy_adherence — Stayed inside your policies

• Pass: every commitment, refund, exception, or claim the assistant makes is allowed by your written policies.
• Fail: The assistant promises, grants, or states something your policies do not allow, or invents a policy.

correct_tool_use — Used the right tools with the right inputs

• Pass: the assistant looks up or changes things only through the tools meant for them, with inputs grounded in what the customer said, and does not repeat a call that already answered.
• Fail: The assistant calls the wrong tool, calls one with invented or mismatched inputs, loops on the same call, or answers without looking up what it needed to.

tone — Tone fits your brand and the customer's state

• Pass: the replies are clear, respectful, and match the voice your team uses, including when the customer is frustrated.
• Fail: A reply is curt, condescending, robotic, or off-brand, or ignores that the customer is upset.

escalated_when_needed — Handed off to a human when it should have

• Pass: a conversation that needs a person (a sensitive case, a request outside the assistant's authority, or a customer who asks for one) is escalated promptly.
• Fail: The assistant keeps going when it should have handed off, or escalates a routine case it could have resolved.

The guide's escalation band starts at epsilon: 0.1: a grade within that distance of the route's pass threshold is treated as judge-uncertain and sent to a human. It is a starting band — tune it as your grader calibrates.

Golden checklist

Roughly twenty examples that cover:

• One clearly resolved conversation per common contact reason — the ordinary case your team handles every day.
• A conversation that went quiet without being solved, so the guide learns that silence is not resolution.
• A conversation where the customer asked for something your policy forbids, handled correctly — and one handled wrongly.
• A tool-heavy conversation (an order or account lookup) with correct inputs, and one where the assistant looped on the same lookup.
• A frustrated customer handled well, so tone is judged on the hard cases rather than the easy ones.
• A sensitive case (legal, safety, billing dispute, or churn risk) that should have been escalated, and a routine case that should not.
• Aim for roughly twenty goldens total, weighted toward the boundaries above rather than the easy centre of each contact reason.

Escalation starter

Start by sending to a human every conversation the assistant escalated, every conversation a customer reopened (post it as an outcome named reopened), and every sensitive conversation regardless of how it ended. The machine-side band lives in starter_guide.escalation.epsilon: a grade within that distance of your conversation task's pass threshold is treated as grader-uncertain and sent to the human queue. The 0.1 here is a starting band — widen it while you build trust, narrow it as your grader calibrates.

Apply it

By hand: wrap each conversation in rig.conversation, turn on content capture for the agent so the transcript is readable, and write the starter criteria into the grading guide of the conversation:<agent> task in your team's words. Grade a batch of past conversations in the console (Conversations → open one → Pass / Fail) until the grader calibrates. Post what happened later — resolved, reopened — with rig.outcome or POST /v1/outcomes; it shows beside the grader as outcome agreement, a receipt that never calibrates the grader on its own.

From a coding agent: set_task_settings (the task-settings scope) authors the grading guide for conversation:<agent> as-is, and record_outcome posts outcomes from your webhook handler. No new tool is involved beyond those. See the use-case templates overview for the full two-path walkthrough.