Specialist-model research desk

Comparisons

How do we normalise competing proposals?

Frontier Development research deskSection 03 · Page 6 of 11Last updated 26 October 2026

Issue a common response schedule with rows for data curation, labels, baseline optimisation, adaptation runs, evaluation, serving integration, security review, monitoring, on-call support, retraining and transition. For each row, ask “included price / usage assumption / buyer owner / supplier owner / excluded work / acceptance evidence”. Keep unknown prices visible.

Example, explicitly hypothetical: Proposal A quotes $30,000 for training; Proposal B quotes $55,000 for training plus integration and evaluation. It is invalid to call A cheaper overall until the missing work is costed. Ask A to price the same scope, or add a transparent buyer estimate with a separate confidence label. Do not silently fill exclusions with zero.

Score only after hard gates. A stronger average cannot compensate for unavailable data rights, a failed critical quality requirement or no acceptable exit. Suggested weighted criteria appear in scorecard.md; weights are a purchasing policy, not a research finding.

TRACE THE EVIDENCE

Section source library

This page contains no direct external citations. These are the section’s references; they do not independently validate the desk’s proposed scoring or methods.

  1. A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented Generation — He, Popa and Danilevsky, 19 October 2026 →arxiv.org
  2. Together AI — AI Native Cloud →www.together.ai
  3. Llama 3.3 Community License Agreement — Meta →github.com
  4. Pricing — Together AI, snapshot checked 26 October 2026 →www.together.ai
  5. Demystifying evals for AI agents — Anthropic, 9 January 2026 →www.anthropic.com
  6. SFT Trainer — Hugging Face TRL documentation →huggingface.co
  7. RouteLLM: Learning to Route LLMs with Preference Data — Ong et al., revision 23 February 2025 →arxiv.org
  8. Extend — Extraction Overview →docs.extend.ai
  9. QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 23 May 2023 →arxiv.org
  10. LoRA Learns Less and Forgets Less — Biderman et al., revision 20 September 2024 →arxiv.org
  11. Quantization — Hugging Face PEFT documentation →huggingface.co
  12. Fireworks — specialised intelligence platform →fireworks.ai
  13. Winder.AI — Generative AI Consulting & Development Services →winder.ai
  14. Modal — platform for production AI →modal.com
  15. Baseten — inference platform →www.baseten.co