Write the acceptance rubric
Specify what makes a whole task acceptable and identify failures that cannot be averaged away. For extraction, that might include a wrong total or missing mandatory field. For support, an apparently helpful answer can still fail because it is unsupported or unsafe. Field-level accuracy and final task acceptance should remain separate metrics.
Create a reviewed calibration set
Have appropriate human reviewers score a bounded sample using the same rubric. Resolve disagreements explicitly and retain ambiguous examples. Compare automated judgements with this reviewed reference, including difficult and minority slices. A high overall agreement figure can hide a poor ability to detect consequential failures.
Version the grader
Record the judge model, prompt, rubric, settings and date with every evaluation. A changed grader can move scores even if the production model does not change. Check for position, verbosity or style effects through controlled tests. Treat automated ratings as evidence from a particular measurement process, not as an independent statement of truth.
Use human review where it matters
Route uncertain or consequential cases for review and periodically recalibrate against new production failures. Estimate the resulting review cost in the business case. If the grader cannot reliably distinguish accepted from rejected tasks, do not use its scores to justify commissioning. The pilot brief makes approval of the scoring policy an explicit gate.
Bring to the decision review
Bring examples of judge-human disagreement, including false acceptance of a serious failure. Ask the review team whether the rubric or the grader should change, and document the choice before rescoring. A change in judgement policy should be visible in the report version. Never compare old and new scores as if the measurement process were unchanged, even when the underlying task and candidate remain identical.
Sources & editorial record
This is Frontier Development’s decision framework. The references below support the general technical concepts; the worked scenarios and procurement questions are our analysis.
References accessed 16 October 2026. No current price, product availability or legal clearance is asserted. Published study findings are attributed above; this guide reports no Frontier Development experiment. Read the comparison method.
Review history and research boundary
Edition 1.1 · 16 October 2026: added claim-level attribution, explicit evidence boundaries and links to the flagship synthesis. Published by Frontier Development. No named external technical reviewer is recorded. This guide provides a decision framework; it is not a Frontier Development experimental result.
Read the primary-study synthesis →Open the pilot tool
Explore a complete synthetic example before using your own inputs.