Prime Collective Communications Library: A Technical Report

The Prime Collective Communications Library (PCCL) is a novel, fault-tolerant communication library specifically engineered for distributed machine learning tasks, particularly over the public internet. It introduces a master-client programming model that supports dynamic peer membership and resilient fault recovery, allowing the system to continue operations even if participants join or fail unexpectedly. PCCL ensures bit-identical state consistency across all peers through parallel hashing and on-demand data transfers, and it optimizes communication pathways by measuring bandwidth and solving the asymmetric traveling salesman problem. The library facilitates efficient distributed training algorithms, such as DiLoCo and its asynchronous variant, which significantly reduce communication overhead by overlapping local computations with global updates. Benchmarks demonstrate PCCL's robustness and efficiency across various network configurations, including cross-continental connections, making it a viable solution for training on dynamic and unreliable networks like spot instances or multi-cloud environments.