MocoSFL: enabling cross-client collaborative self-supervised learning
Authors: Jingtao Li, Lingjuan Lyu, Daisuke Iso, Chaitali Chakrabarti, Michael Spranger
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We simulate the multi-client Moco SFL scheme on a Linux machine, where we use different CPU threads to simulate different clients and a single RTX-3090 GPU to simulate the cloud server. We use CIFAR-10 as the main dataset and also present results on CIFAR-100 and Image Net 12-class subset as in Li et al. (2021). For accuracy performance evaluation, we adopt similar linear probe methods as in Grill et al. (2020); Zhuang et al. (2022). |
| Researcher Affiliation | Collaboration | Jingtao Li Arizona State University jingtao1@asu.edu Lingjuan Lyu Sony AI lingjuan.lv@sony.com Daisuke Iso Sony AI daisuke.iso@sony.com Chaitali Chakrabarti Arizona State University chaitali@asu.edu Michael Spranger Sony AI michael.spranger@sony.com |
| Pseudocode | No | The paper includes a system diagram (Figure 2) but no structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code is available at https://github.com/Sony AI/Moco SFL. |
| Open Datasets | Yes | We use CIFAR-10 as the main dataset and also present results on CIFAR-100 and Image Net 12-class subset as in Li et al. (2021). |
| Dataset Splits | Yes | For the IID case, we assume the entire dataset is divided randomly and equally among all clients. For non-IID experiments, we mainly consider the pathological (aka. class-wise) non-IID distribution as in Mc Mahan et al. (2017); Zhuang et al. (2022) where we assign 2 classes of CIFAR-10/Image Net-12 data or 20 classes of CIFAR-100 data randomly to each client. |
| Hardware Specification | Yes | We simulate the multi-client Moco SFL scheme on a Linux machine, where we use different CPU threads to simulate different clients and a single RTX-3090 GPU to simulate the cloud server. On a Raspberry Pi 4B device, the Moco SFL-based scheme requires less than 1MB of memory and less than 40MB of communication, and consumes less than 5W power. |
| Software Dependencies | No | To evaluate overhead, we use commonly-used libraries such as fvcore for FLOPs and torch.cuda.memory allocated for memory (please see Appendix A.5 for detail.). No specific version numbers for these libraries or other critical software components are provided. |
| Experiment Setup | Yes | We perform Moco SFL training for a total of 200 epochs, using SGD as the optimizer with an initial learning rate of 0.06. For the hyper-parameter choices, we follow two principles introduced in Appendix A.1 we let each client use a batch size of 1 and use the synchronization frequency of f S = (1000/NC)/epoch, and we set the client sampling ratio to 100/NC to keep the same equivalent batch size at the server end. |