How LLM Inference Works: KV Cache, Batching & Speculative Decoding

This Stanford lecture explores the mechanics of LLM inference, explaining why generating text is heavily memory-bound compared to model training. The lecture breaks down key performance metrics, arithmetic intensity, the function of the KV cache during decoding, and optimization techniques such as speculative decoding and continuous batching. Open article →