Specialist-model research desk

Scorecard

How do we score each criterion?

Frontier Development research deskSection 09 · Page 3 of 13Last updated 26 October 2026

Use U for unknown and 0–4 for assessed performance:

Rating Meaning Typical evidence
U Not assessable Missing answer, inaccessible evidence or unclear scope
0 Does not meet the criterion Inspected evidence shows an unacceptable gap
1 Assertion or weak plan Marketing statement, generic demo or incomplete commitment
2 Partly substantiated Relevant material exists but important elements remain unproven
3 Meets the requirement Buyer has inspected relevant, adequate evidence
4 Strongly exceeds the requirement Reproducible, relevant evidence and a demonstrated advantage beyond the minimum

For a conservative total, U contributes zero points but remains labelled U, distinct from a demonstrated failure. Record source, date, evaluator and unresolved question for every rating. Keep the two evaluators’ initial scores before reconciliation so disagreement is visible.

Weighted contribution = criterion weight × rating / 4. Sum contributions for a score out of 100. For U, substitute zero only in the arithmetic, not in the interpretation.

TRACE THE EVIDENCE

Section source library

This page contains no direct external citations. These are the section’s references; they do not independently validate the desk’s proposed scoring or methods.

  1. Together AI — AI Native Cloud →www.together.ai
  2. AI Risk Management Framework — NIST →www.nist.gov
  3. Demystifying evals for AI agents — Anthropic, 9 January 2026 →www.anthropic.com
  4. GroupKFold — scikit-learn documentation →scikit-learn.org
  5. A Field Guide to Rapidly Improving AI Products — Hamel Husain →hamel.dev
  6. Llama 3.3 Community License Agreement — Meta →github.com
  7. Pricing — Together AI, snapshot checked 26 October 2026 →www.together.ai