Template: call disposition QA

From a call transcript, check the disposition the agent logged and score the call against your QA scorecard, handing off the disputed ones to a supervisor. This page is one of the use-case templates — a starting point, not a finished route. ModelRig takes the transcript as text, so you bring your own transcription upstream; the template does not read audio. The machine-readable source is node_modules/modelrig/templates/call-disposition-qa.json.

Does this task fit?

Call disposition QA fits the shipped teach → calibrate → grade → optimize loop when four conditions hold. (1) Your dispositions and scorecard are stable enough that last month's calls still describe this month's — if you are still redrawing the scorecard, settle it before you calibrate. (2) Your QA team can write down what a correct disposition and a correct score are, which is the grading guide below. (3) An experienced QA reviewer can grade a batch of past calls against the calls they would have scored, so a grader can be calibrated against them. (4) You handle enough calls that calibration and optimization pay back. This template assumes the call arrives as text: ModelRig takes a transcript as input, so you bring your own transcription upstream and feed the transcript here — the template does not read audio. It gives you a disposition shell and a scorecard-shaped guide to fill in; it makes no claim about how well any model scores a call — that is what your goldens and grades measure.

Starter schema

Replace every TODO with your own taxonomy; the enum slots ship without values on purpose, so an unfilled template refuses to look finished.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$comment": "STARTER SCHEMA. Replace every TODO with your own dispositions and scorecard before use. The template runs as-is: free-text fields validate immediately. TODO markers show where adding your own enum values tightens grading to your process; do not copy example words from the docs.",
  "title": "CallDispositionQa",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "disposition",
    "needs_human"
  ],
  "properties": {
    "disposition": {
      "type": "string",
      "description": "The disposition the transcript actually supports for this call. TODO(enum): constrain this to YOUR own disposition labels by adding an \"enum\": [ ... ] here. It runs as free text immediately; add your own enum values here to tighten validation and grading to your process.",
      "$comment": "TODO(enum): add your call dispositions as the enum for this field."
    },
    "scorecard": {
      "type": "object",
      "description": "The QA scorecard result. TODO: replace this placeholder with one named property per item on YOUR scorecard (e.g. verified_identity, disclosed_recording, resolved_issue), each with its own type and description. Keep additionalProperties false once you have listed them so nothing extra is invented.",
      "additionalProperties": true,
      "$comment": "TODO(fields): replace with your real scorecard items; add a required list once you know which items must always be present."
    },
    "matches_agent_log": {
      "type": "boolean",
      "description": "True when the disposition the transcript supports matches the one the agent logged. Lets you surface logged-versus-actual mismatches. Optional; delete it if you do not track the agent's own log."
    },
    "needs_human": {
      "type": "boolean",
      "description": "True when the call is disputed, sensitive, or the transcript is too incomplete to score confidently and a supervisor should review. Pairs with the escalation band in the grading guide."
    }
  }
}

Starter grading guide

The starter criteria run as written — sharpen each one in your team's own words as you grade. They are in the exact jsonb shape the set_task_settings write path validates — binary checks, each with a one-sentence pass and fail. There is no pass_threshold here: that lives in the route's grade: block.

disposition-correct — Disposition matches the call

• Pass: the disposition is the one your QA team would assign given what the transcript shows happened on the call.
• Fail: The disposition does not match the call — a resolved outcome on a call that escalated, or the reverse.

grounded-in-transcript — Every judgment is grounded in the transcript

• Pass: each scorecard result cites something actually said on the call; nothing is inferred beyond the transcript.
• Fail: The output asserts an event, a phrase, or an outcome that the transcript does not contain.

scorecard-applied-consistently — Scorecard is applied by your rules

• Pass: each scorecard item is judged the way your rubric defines it, not by a general impression of the call.
• Fail: A scorecard item is marked pass or fail in a way your own rubric would not, or is skipped when it applies.

human-flag-when-unsure — Disputed or sensitive calls are flagged for a human

• Pass: a call that is disputed, compliance-sensitive, or has a partial or unclear transcript sets needs_human rather than being auto-scored.
• Fail: A call a supervisor would have reviewed is scored automatically with false confidence.

The guide's escalation band starts at epsilon: 0.1: a grade within that distance of the route's pass threshold is treated as judge-uncertain and sent to a human. It is a starting band — tune it as your grader calibrates.

Golden checklist

Roughly twenty examples that cover:

• One clear example per disposition in your process — the ordinary, unambiguous call for each outcome.
• A call that could plausibly take two dispositions — the golden fixes which one wins and why.
• A call where the agent's logged disposition does not match what the transcript shows — the golden records the mismatch.
• A compliance-sensitive call (a required disclosure, a dispute, a vulnerable caller) that should reach a supervisor regardless of score.
• A clean, by-the-book call — so the guide learns the ordinary pass, not only the failures.
• An edge case: a partial, cut-off, or garbled transcript — the golden shows it flagged for a human rather than scored as if complete.
• Aim for roughly twenty goldens total, weighted toward the disputed, mismatched, and sensitive calls rather than the clean centre.

Escalation starter

Start by escalating to a supervisor whenever the model sets needs_human, and always for compliance-sensitive calls regardless of the score. The machine-side band lives in starter_guide.escalation.epsilon: a grade within that distance of your route's pass threshold is treated as judge-uncertain and sent to the supervisor queue. The 0.1 here is a starting band — keep it wide until an experienced reviewer has graded enough calls to trust the grader, then narrow it.

Apply it

By hand: replace the placeholder scorecard object with one named property per item on your scorecard, each with its own type; replace the TODO disposition list with your own labels; sharpen each starter criterion in your team's words; gather the goldens the checklist describes; set the escalation band.

From a coding agent: the template feeds through the tools that already exist — create_route drafts the route bundle from the starter schema (files only, never your source), and set_task_settings (the task-settings scope) authors the grading guide as-is — the starter criteria run as written; sharpen them to your own rules as you grade. No new tool is involved. See the use-case templates overview for the full two-path walkthrough.