Listen/watch: Latent Space episode page with transcript and YouTube link. Published 3 August 2026; 101 minutes.
Who: Baseten’s Philip Kiely and Ali Taha, as introduced in the episode. Topics include cache-aware routing, prefill/decode, quantisation, speculative decoding and differences between public APIs and dedicated deployments.
Useful takeaway: running the same weights does not by itself establish equivalent service performance. The discussion is valuable for asking what changed in the runtime and which workload was used to measure an improvement. Vendor-specific speedups are not adopted here as buyer savings.
Buyer exercise — proposed: draft a serving test matrix with prompt-length buckets, output lengths, concurrency, warm/cold conditions, quantisation and fallback policy. Ask for end-to-end request completion as well as token-generation measurements. Keep the matrix small enough to execute but broad enough to cover the important production shapes.
Question to take into procurement: “Which latency and quality results come from the configuration you will actually sell us?”