Specialist-model research desk

Scorecard

Can we see a worked example?

Frontier Development research deskSection 09 · Page 7 of 13Last updated 26 October 2026

Consider two fictional plans, not real suppliers. Plan A scores 84 but refuses a buyer-controlled holdout: it fails the independent-acceptance gate and cannot be preferred on its total. Plan B scores 74 with all pilot gates resolved and 90% evidence coverage: it is a conditional candidate for a bounded pilot, with remediation tied to its weakest criteria.

A separate arithmetic example: a 20-weight criterion rated 3 contributes 15 points; a 15-weight criterion rated 2 contributes 7.5. The supplier does not receive full weight merely for providing a document. Evidence must support the rating, and unknowns must not be disguised as compliant.

TRACE THE EVIDENCE

Section source library

This page contains no direct external citations. These are the section’s references; they do not independently validate the desk’s proposed scoring or methods.

  1. Together AI — AI Native Cloud →www.together.ai
  2. AI Risk Management Framework — NIST →www.nist.gov
  3. Demystifying evals for AI agents — Anthropic, 9 January 2026 →www.anthropic.com
  4. GroupKFold — scikit-learn documentation →scikit-learn.org
  5. A Field Guide to Rapidly Improving AI Products — Hamel Husain →hamel.dev
  6. Llama 3.3 Community License Agreement — Meta →github.com
  7. Pricing — Together AI, snapshot checked 26 October 2026 →www.together.ai