Error Feedback Reloaded: From Quadratic to Arithmetic Mean of Smoothness Constants
Authors: Peter Richtárik, Elnur Gasanov, Konstantin Pavlovich Burlachenko
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, we validate our theoretical findings with suitable experiments. |
| Researcher Affiliation | Academia | Peter Richt arik AI Initiative KAUST , Saudi Arabia; Elnur Gasanov AI Initiative KAUST, Saudi Arabia; Konstantin Burlachenko AI Initiative KAUST, Saudi Arabia |
| Pseudocode | Yes | Algorithm 1 EF21: Error Feedback 2021 |
| Open Source Code | No | If you are interested in the source code for all experiments, please contact the authors. |
| Open Datasets | Yes | We utilized six datasets from LIBSVM (Chang & Lin, 2011). |
| Dataset Splits | Yes | The dataset shuffling strategy, detailed in Appendix I.5, was employed to emulate heterogeneous data distribution. Each client was assigned the same number of data points. |
| Hardware Specification | Yes | We carried out experiments on a compute node with Ubuntu 18.04 LTS, 256 GBytes of DRAM DDR4 memory at 2.9GHz, and 48 cores (2 sockets with 24 cores per socket) of Intel(R) Xeon(R) Gold 6246 CPU at 3.3GHz. All our computations were carried on CPU. |
| Software Dependencies | No | We used the Python software suite FL Py Torch (Burlachenko et al., 2021) to simulate the distributed environment for training. |
| Experiment Setup | Yes | The initial gradient estimators were chosen as g0 i = fi(x0) for all i [n]." and "The coefficient λ for (b) (f) is set to 0.001, and for (a) is set to 1, 000 for numerical stability." and "The number of clients n is 1, 000." and "The step size for EF21 is set according to (Richt arik et al., 2021), and the step size for EF21-W is set according to Theorem 3." |