Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data
Authors: Sai Aparna Aketi, Abolfazl Hashemi, Kaushik Roy
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, and Image Nette), model architectures, and network topologies. Our experiments show that the proposed method achieves state-of-the-art performance for decentralized learning on heterogeneous data via a 1 6% improvement in test accuracy compared to other existing techniques. |
| Researcher Affiliation | Academia | Sai Aparna Aketi Abolfazl Hashemi Kaushik Roy Department of Electrical and Computer Engineering Purdue University West Lafayette, IN 47906 {saketi, abolfazl, kaushik}@purdue.edu |
| Pseudocode | Yes | Algorithm 1 Global Update Tracking (GUT) Input: Each agent i [1, n] initializes model parameters x0 i and neighbors copy ˆx0 j, step size η, GUT scaling factor µ, mixing matrix W = [wij]i,j [1,n], N(i) represents neighbors of i including itself, and note ˆxt i = xt i. Each agent simultaneously implements the TRAIN( ) procedure |
| Open Source Code | Yes | The source code is available at https://github.com/aparna-aketi/ global_update_tracking |
| Open Datasets | Yes | We present the analysis on (a) Datasets: CIFAR-10, CIFAR-100, Fashion MNIST, and Imagenette. |
| Dataset Splits | No | No explicit mention of training/validation/test dataset splits (e.g., '80/10/10 split', '70% training, 15% validation, 15% test') or cross-validation setup was found. |
| Hardware Specification | Yes | All our experiments were conducted on a system with Nvidia GTX 1080ti card with 4 GPUs except for Image Nette simulations. We used NVIDIA A40 card with 4 GPUs for Image Nette simulations. |
| Software Dependencies | No | Algorithm 3 Global Update Tracking with momentum (QG-GUTm) Pytorch Implementation (mentions PyTorch, but no version number). No specific version numbers for software dependencies were found. |
| Experiment Setup | Yes | All the experiments have the stopping criteria set to 200 epochs. The initial learning rate is set to 0.1. We decay the step size by 10 in multiple steps at 100th and 150th epoch. Table 6 presents values of the scaling factor µ used in the experiments. For all the experiments, we use a mini-batch size of 32 per agent. The stopping criteria is a fixed number of epochs. We have used a momentum of 0.9 for all QG-DSDm and QG-GUTm experiments. |