Specialist-model research desk

Do-it-yourself game plans

Stage 1: define a task and an economic hurdle

Frontier Development research deskSection 02 · Page 3 of 15Last updated 26 October 2026

Planning allowance: 2–4 working days. Select one workflow and write down who receives the result, what action follows and what constitutes an unacceptable error. Exclude new features that are not necessary to test the decision.

For extraction, define field names, treatment of missing or illegible values, currency/date conventions and business rules. For support, distinguish drafting from taking an action; specify permissible actions and required escalation. Assign a cost or operational consequence to each important error without pretending every harm can be priced.

Collect a representative billing period and compute current accepted output. Separate an API request from a task: a task may involve several calls or retries. The proposed economic rule is “meet all critical requirements and improve the chosen whole-life measure under the approved downside case”, not simply “reduce tokens”.

Exit artefacts: task contract, metric dictionary, current monthly cost ledger, named decision owner and explicit no-build outcome.

TRACE THE EVIDENCE

Section source library

This page contains no direct external citations. These are the section’s references; they do not independently validate the desk’s proposed scoring or methods.

  1. Demystifying evals for AI agents — Anthropic, 9 January 2026 →www.anthropic.com
  2. Quantization — Hugging Face PEFT documentation →huggingface.co
  3. LoRA Learns Less and Forgets Less — Biderman et al., revision 20 September 2024 →arxiv.org
  4. Pricing — Together AI, snapshot checked 26 October 2026 →www.together.ai
  5. GroupKFold — scikit-learn documentation →scikit-learn.org
  6. A Field Guide to Rapidly Improving AI Products — Hamel Husain →hamel.dev
  7. Guidance on HIPAA & Cloud Computing — HHS Office for Civil Rights →www.hhs.gov
  8. TimeSeriesSplit — scikit-learn documentation →scikit-learn.org
  9. Extend — Extraction Overview →docs.extend.ai
  10. SFT Trainer — Hugging Face TRL documentation →huggingface.co
  11. RouteLLM: Learning to Route LLMs with Preference Data — Ong et al., revision 23 February 2025 →arxiv.org