Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
Authors: Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, Michal Valko
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | BYOL reaches 74.3% top-1 classification accuracy on Image Net using a linear evaluation with a Res Net-50 architecture and 79.6% with a larger Res Net. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. |
| Researcher Affiliation | Collaboration | 1DeepMind 2Imperial College |
| Pseudocode | Yes | python pseudo-code based on the libraries JAX [64] and Haiku [65] is provided in in Appendix J. |
| Open Source Code | Yes | Our implementation and pretrained models are given on Git Hub.3 |
| Open Datasets | Yes | We assess the performance of BYOL s representation after self-supervised pretraining on the training set of the Image Net ILSVRC-2012 dataset [21]. |
| Dataset Splits | Yes | We follow the semi-supervised protocol of [74, 76, 8, 32] detailed in Appendix D.1, and use the same fixed splits of respectively 1% and 10% of Image Net labeled training data as in [8]. |
| Hardware Specification | Yes | We run ablations over 300 epochs on 64 TPU v3 cores... |
| Software Dependencies | No | The paper mentions the use of 'JAX [64] and Haiku [65]' libraries for its pseudocode, but it does not specify exact version numbers for these software dependencies or any other critical software components used in the experiments. |
| Experiment Setup | Yes | For all the experiments in this section, we set the initial learning rate to 0.3 with batch size 4096, the weight decay to 10 6 as in Sim CLR [8] and the base target decay rate τbase to 0.99. |