Define success at the business-task level, with explicit failure constraints. Averages alone can hide rare but costly errors.
| Workflow | Primary acceptance measure | Essential companion measures |
|---|---|---|
| Invoice extraction | Entire document satisfies agreed field and business rules | Critical-field errors, abstention, review minutes, duplicate handling |
| Support classification | Correct destination and permitted next action | Rare-intent recall, wrong escalation, tenant-policy errors |
| Support drafting | Reviewer accepts answer under the rubric | Unsupported statements, edits required, review time |
| Autonomous resolution | Correct permitted outcome completed | Unauthorised actions, reversals, unresolved cases, escalation quality |
Show denominators, sample composition and uncertainty. For correlated records, estimate uncertainty at a suitable cluster level; ask an evaluation specialist to check the design if the decision turns on a small difference. Anthropic’s evaluation guidance distinguishes code, model and human graders. The desk’s recommendation is to use deterministic checks where possible and calibrate subjective grading against adjudicated examples.