Specialist-model research desk

Scorecard

How do we measure evidence coverage separately?

Frontier Development research deskSection 09 · Page 5 of 13Last updated 26 October 2026

Add an inspected-evidence flag for each criterion. Set it to 1 only when the buyer has actually reviewed material directly relevant to that criterion; a promise to send it later is 0. Weighted coverage is the sum of weights with flag 1, regardless of whether the evidence is favourable.

The desk’s suggested ranking rule is: all applicable hard gates resolved, no U on a critical criterion, and at least 80% weighted evidence coverage before using totals to prefer one proposal. This 80% threshold is an editorial policy, not a statistically derived confidence level. If coverage is lower, issue clarification requests rather than label the supplier technically inferior.

TRACE THE EVIDENCE

Section source library

This page contains no direct external citations. These are the section’s references; they do not independently validate the desk’s proposed scoring or methods.

  1. Together AI — AI Native Cloud →www.together.ai
  2. AI Risk Management Framework — NIST →www.nist.gov
  3. Demystifying evals for AI agents — Anthropic, 9 January 2026 →www.anthropic.com
  4. GroupKFold — scikit-learn documentation →scikit-learn.org
  5. A Field Guide to Rapidly Improving AI Products — Hamel Husain →hamel.dev
  6. Llama 3.3 Community License Agreement — Meta →github.com
  7. Pricing — Together AI, snapshot checked 26 October 2026 →www.together.ai