llm-d integrates SGLang as Kubernetes inference engine
Contributors from the llm-d and SGLang communities have integrated SGLang as a first-class, Kubernetes-native inference engine in llm-d, enabling Wide-EP deployments and tiered KV-cache offloading for long-context agentic workloads. Offloading evicted KV blocks to host RAM or shared storage maintains high cache hit rates, improves concurrency performance, and provides resilience during pod failovers and cold restarts. Open article →