Simplicial Embeddings in Self-Supervised Learning and Downstream Classification
Authors: Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, Aaron Courville
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR100 and Image Net. Finally, when used in a downstream classification task, we show that SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes. |
| Researcher Affiliation | Academia | Mila, Université de Montréal, National University of Singapore, CIFAR Fellow |
| Pseudocode | No | The paper describes the formal re-normalization process in text and equations (e.g., Equation 1), and provides diagrams like Figure 2a, but it does not include a distinct block labeled "Pseudocode" or "Algorithm". |
| Open Source Code | Yes | The code for reproducing the results is available at: https://github.com/lavoiems/simplicial-embeddings/. |
| Open Datasets | Yes | natural image datasets such as CIFAR100 and Image Net. ... CIFAR-100 (Krizhevsky, 2009). ... IMAGENET (Deng et al., 2009). |
| Dataset Splits | Yes | The SEM hyper-parameters are selected by using a validation set of 10% of the training set of CIFAR-100 and 10 samples per class on the in distribution dataset for IMAGENET. |
| Hardware Specification | Yes | For all our CIFAR-100 training, we used 1 RTX-8000 per experiment. For our Image Net experiments, we used parallel training with 2 40GB A100 for the training with Res Net50 and Res Net50-x2 and 4 40GB A100 for the training with Res Net50-x4. |
| Software Dependencies | No | The paper lists software projects that contributed to the work, such as "Pytorch (Paszke et al., 2019), Orion (Bouthillier et al., 2022), Solo-Learn (da Costa et al., 2021), Scikit-Learn (Pedregosa et al., 2011), and Numpy (Harris et al., 2020)." However, it does not specify the exact version numbers (e.g., PyTorch 1.x, NumPy 1.x) for these dependencies. |
| Experiment Setup | Yes | Training setup. For all experiments, we build off the implementation of the baseline models from the Solo-Learn library (da Costa et al., 2021). We probe the encoder s output for the baseline methods, as typically done in the literature. For models with SEM, we probe the SEM normalized representation (i.e. ˆz). In our experiments, the embedder is a linear layer followed by Batch Norm (Ioffe & Szegedy, 2015). Unless mentioned otherwise, we use L = 5000 and V = 13 for the SEM representation. We do not perform any search for the non-SEM hyper-parameters. The SEM hyper-parameters are selected by using a validation set of 10% of the training set of CIFAR-100 and 10 samples per class on the in distribution dataset for IMAGENET. The test accuracy is obtained by retraining the model with all of the training data using the parameters found with the validation set. We pre-train the SSL models for 200 epochs on IMAGENET and 1000 epochs on CIFAR-100. |