Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Homomorphic Self-Supervised Learning
Authors: T. Anderson Keller, Xavier Suau, Luca Zappella
TMLR 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this work, we observe that many existing self-supervised learning algorithms can be both unified and generalized when seen through the lens of equivariant representations. Specifically, we introduce a general framework we call Homomorphic Self-Supervised Learning, and theoretically show how it may subsume the use of input-augmentations provided an augmentation-homomorphic feature extractor. We validate this theory experimentally for simple augmentations, demonstrate the necessity of representational structure for feature-space SSL, and further empirically explore how the parameters of this framework relate to those of traditional augmentation-based self-supervised learning. |
| Researcher Affiliation | Industry | T. Anderson Keller EMAIL Apple Xavier Suau EMAIL Apple Luca Zappella EMAIL Apple |
| Pseudocode | No | The paper describes methodologies and processes using mathematical equations and textual descriptions, but does not include any clearly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | No | The paper references third-party code like SESN's GitHub repository for architectural details (e.g., 'https://github.com/ISosnovik/sesn/blob/master/models/stl_ses.py#L19'), but there is no explicit statement from the authors about releasing their own code for the specific methodology described in this paper. |
| Open Datasets | Yes | Table 1: MNIST (Le Cun & Cortes, 2010), CIFAR10 (Krizhevsky et al.) and Tiny Image Net (Le & Yang, 2015) top-1 test accuracy (mean std. over 3 runs) of a detached classifier trained on the representations from SSL methods with different backbones. |
| Dataset Splits | Yes | Table 1: MNIST (Le Cun & Cortes, 2010), CIFAR10 (Krizhevsky et al.) and Tiny Image Net (Le & Yang, 2015) top-1 test accuracy (mean std. over 3 runs) of a detached classifier trained on the representations from SSL methods with different backbones. |
| Hardware Specification | No | On average each MNIST run took 1 hour to complete distributed across 8 GPUs, and each CIFAR10/TIN run took 10 hours to complete distributed across 64 GPUs. In total this amounted to roughly 85,000 GPU hours. |
| Software Dependencies | No | All models presented in this paper are built using the convolutional layers from the SESN (Sosnovik et al., 2020) library for consistency and comparability. For training, we use the LARS optimizer (You et al., 2017) with an initial learning rate of 0.1, and a batch size of 4096 for all models. |
| Experiment Setup | Yes | Training Details For training, we use the LARS optimizer (You et al., 2017) with an initial learning rate of 0.1, and a batch size of 4096 for all models. We use an NCE temperature (τ) of 0.1, half-precision training, a learning rate warm-up of 10 epochs, a cosine lr-update schedule, and weight decay of 1 10 4. On MNIST we train for 500 epochs and on CIFAR10 and Tiny Imag Net (TIN) we train for 1300 epochs. |