DP, TP, PP and EP: How LLM Parallelism Actually Fits Together
The author examines how data, tensor, pipeline, and expert parallelism combine in large language models, providing a detailed case study of serving a 35-billion-parameter mixture-of-experts model on eight GPUs. The analysis demonstrates how expert parallelism reuses tensor and data parallelism groups to change memory distribution and GEMM shapes without reducing total expert weights per GPU. The author explains that combining expert parallelism with DP-attention reduces KV cache duplication and enables larger batches for better throughput. Open article →