Google DeepMind streams DiLoCo in MaxText

developers.googleblog.com5 min read

Google DeepMind and Google Research developers have introduced Streaming DiLoCo, a distributed optimization algorithm implemented in MaxText that enables efficient LLM training across geographically distributed datacenters and wide-area networks. By partitioning model parameters into disjoint fragments and synchronizing them incrementally over inner-loop steps, the method eliminates periodic network stalls, matches or improves validation loss, and decouples large-scale pre-training from monolithic high-bandwidth compute clusters. Open article →