The Hitchhiker’s Guide to Training on Compute Islands | Core Automation

The author presents a practical case study on training a 32-billion-parameter mixture-of-experts model across two separate clusters located 9,000 miles apart using different GPU architectures. By combining fragment-wise synchronization, stochastic rounding with error feedback, and butterfly mixing to replace traditional all-reduce operations, the two-site run achieved slightly better evaluation performance than standard training. This approach demonstrates that scattered compute islands can be effectively utilized for frontier AI training without requiring a single colocated cluster. Open article →