Why it matters: an automated score can agree with a model’s preferred style while missing unacceptable factual or operational errors.
A strong answer should cover: adjudicated reference cases, false acceptance, false rejection, ambiguous labels, judge/model version changes and human-review cost. Anthropic’s evaluation guidance explicitly discusses grader calibration.
Follow-up: “Show a case where the judge and reviewer disagree; what would change in the rubric or system?” Ask for uncertainty and unresolved disagreement to remain visible, not be averaged away.
Interview status: Not yet interviewed.