| Criterion | Weight | What good looks like | What weak or bad looks like | Evidence requested |
|---|---|---|---|---|
| Task quality and failure coverage | 20 | Acceptance tied to business outcomes; difficult slices and critical errors visible | Leaderboard or cherry-picked demo; failures omitted | Paired results, denominators, errors and slice reports |
| Evaluation integrity | 15 | Buyer-controlled holdout, justified split, calibrated graders and reproducibility | Training/test overlap; score without rubric; supplier controls all answers | Split manifest, access policy, run config and adjudication record |
| Data and adaptation plan | 15 | Rights, labels, lineage, base choice and bounded experiments connect to failures | More data or more training proposed without diagnosis | Inventory, label guide, provenance and ablation plan |
| Whole-life economics | 10 | Common accepted-task denominator; one-off/recurring costs and downside cases | Token saving only; missing labour, retries or minimum commitments | Itemised quote and scenario workbook |
| Production engineering | 10 | Actual serving config, load tests, fallback, rollback and observability | Notebook handover with deployment left implicit | Architecture, traces, operational runbook |
| Security and governance | 10 | Appropriate handling, access, retention, incident and change controls | Generic badges with no product/contract scope | Data-flow review and contractual controls |
| Rights and exit | 10 | Artefact schedule and feasible export/redeployment path | “You own it” without transferable files or permissions | Licence list, export package and rehearsal plan/result |
| Delivery capacity and scope | 5 | Named team, dependencies, milestones and change control | Unnamed resources and vague acceptance | Work plan, responsibility matrix and reference questions |
| Maintenance and reassessment | 5 | Owners, cost, regression suite and triggers to retain/replace | Automatic retraining retainer without diagnosis | Maintenance schedule and baseline refresh policy |
| Total | 100 |
The weighting intentionally favours task evidence over presentation and price. A privacy-sensitive deployment may require different weights, but change them before opening bids. Error analysis and appropriate split design are useful evidence checks; they are not substitutes for inspecting the actual submission.