Specialist-model research desk

Expert Q&A

4. “What would convince you that our holdout is contaminated?”

Frontier Development research deskSection 06 · Page 6 of 17Last updated 26 October 2026

Why it matters: seemingly independent rows can contain the same customer, template or derived answer.

A strong answer should cover: source grouping, near-duplicates, temporal ordering, repeated tuning, synthetic lineage and supplier access. Ask for an inspection procedure and what it cannot detect. Group-based splitting is a mechanism, not a cleanliness certificate.

Follow-up: “Would you redesign the test for new customers, future traffic or unseen templates—and why?” Record which commercial claim each split can support.

Interview status: Not yet interviewed.

TRACE THE EVIDENCE

Sources cited on this page

  1. Group-based splitting →scikit-learn.org