Listen/watch: Latent Space episode page with audio, transcript and video link. Historical episode: 11 October 2024, 117 minutes.
Who: Ankur Goyal, introduced in the episode through his Braintrust work. The page frames evaluation as central to systematic AI product development, rather than an optional step after prompt management.
Useful takeaway: create a repeatable way to compare changes against product expectations. This is most relevant when a team has many promising experiments but cannot explain which one improves a real customer task. Treat company funding and market observations in the recording as historical, not current facts.
Buyer exercise — proposed: choose five sanitised failure patterns from your own product and write what acceptance means for each. Identify which checks are deterministic, which need a domain reviewer and which could use a calibrated model grader. This small exercise is diagnostic, not a statistically sufficient benchmark.
Question to take into procurement: “Show how a failed production case becomes a regression test, and who approves its expected answer.”