Planning allowance: 3–7 working days initially, then review effort as needed. Choose the generalisation question first: new customers, new document layouts, a future period or a changed policy. Construct the split around that question.
Keep related records together and exclude near-duplicates. GroupKFold supports group separation, while TimeSeriesSplit addresses ordered observations; neither automatically solves semantic leakage or chooses the business grouping. Preserve split manifests and hashes with access controls.
Create three sets with distinct purposes: training for learning; development for prompt, rank and threshold choices; final holdout for the award decision. Protect the final answers from suppliers and from synthetic-data generation. A separate realistic stress set can test failure modes, but report it separately from prevalence-weighted production estimates.
Write a scoring guide with accepted, rejected, abstained and escalated outcomes. Use exact checks for structure and business rules; reserve human judgement for genuine ambiguity. The January 2026 Anthropic evaluation guide is useful background for grader design. Determine sample size from the minimum useful difference, failure rates and dependence structure; this document deliberately supplies no universal count.