The Privacy Power of Correlated Noise in Decentralized Learning
Authors: Youssef Allouah, Anastasia Koloskova, Aymane El Firdoussi, Martin Jaggi, Rachid Guerraoui
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we empirically show that DECOR achieves a privacy-utility trade-off matching the CDP baseline, and surpassing the LDP baseline. Recall that LDP is the strongest threat model in decentralized learning, while CDP is the weakest, and thus they represent lower and upper bounds in terms of performance. We compare these algorithms on three strongly convex and non-convex tasks with synthetic and real-world data and multiple network topologies. |
| Researcher Affiliation | Academia | Youssef Allouah 1 Anastasia Koloskova 1 Aymane El Firdoussi 1 Martin Jaggi 1 Rachid Guerraoui 1 1EPFL, Switzerland. |
| Pseudocode | Yes | Algorithm 1 DECOR: DECENTRALIZED SGD WITH CORRELATED NOISE Algorithm 2 SINGLE-STEP SECRDP ACCOUNTANT Algorithm 3 GENERAL SECRDP ACCOUNTANT FOR DECOR |
| Open Source Code | Yes | 1Our code is available at https://github.com/elfirdoussilab1/DECOR. |
| Open Datasets | Yes | We study two strongly convex tasks: least-squares regression on synthetic data and regularized logistic regression on the a9a LIBSVM dataset (Chang & Lin, 2011). ... We consider the MNIST (Le Cun & Cortes, 2010) task with a one-hidden-layer neural network. |
| Dataset Splits | No | The paper does not explicitly state training, validation, and test dataset splits (e.g., percentages or counts). It mentions distributing data among users and using specific datasets but no details on how they were partitioned for training, validation, or testing. |
| Hardware Specification | No | The paper does not specify any hardware details such as GPU models, CPU types, or memory used for running the experiments. It only mentions running experiments with 'n = 16 users'. |
| Software Dependencies | No | The paper mentions that 'Our code is available at https://github.com/elfirdoussilab1/DECOR.' but does not list specific software dependencies with version numbers (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | Setup. We consider n = 16 users on three usual network topologies in increasing connectivity: ring, grid (2d torus), and fully-connected. We use the Metropolis-Hastings (Boyd et al., 2006) mixing matrix... We tune all hyperparameters for each algorithm individually, and run each experiment with four seeds for reproducibility. We account for the privacy budget using our Sec LDP privacy accountant (Algorithm 2). ... For a pre-specified Sec LDP privacy budget ε, we would like to find a corresponding couple of privacy noises (σcdp, σcor)... For our tuning, we choose a grid of learning rates and clipping thresholds. |