Commission adaptation only when a buyer-controlled test shows a valuable gap that a practical API baseline cannot close. Start by identifying the failure, because missing knowledge, an unreliable output format, excessive review and slow serving require different experiments.
For a document product, inspect the full path from page rendering to accepted business record. If the model never sees a faint invoice number, training the answer generator may address the wrong component. For support software, distinguish finding the current refund policy from following that policy, choosing a tool and escalating an uncertain case. These are separate failure categories and should have separate owners.
The desk recommends an initial error ledger: input reference, expected result, actual result, failure class, commercial consequence and proposed intervention. The practitioner case for starting with error analysis is set out in Hamel Husain’s field guide. The table below is a proposed triage method, not an empirical ranking.
| What is failing? | Cheapest discriminating test | Evidence required before adaptation |
|---|---|---|
| Facts are stale or inaccessible | Retrieve current, authorised material | Retrieval still leaves a repeated behavioural failure |
| JSON parses but values are wrong | Inspect source coverage and field errors | Correct source is present, but extraction remains inconsistent |
| Format varies across similar requests | Tighten schema and instructions | A residual pattern persists across representative cases |
| Expensive model handles every request | Test caching, a smaller base model and routing | Quality survives on the eligible traffic slice |
| Domain language is misunderstood | Compare prompting with curated examples | A stable task and trustworthy labels exist |
| Customer requires a private environment | Test permitted deployment options | The requirement cannot be met by the current service contract |