Modulate Your Spectrum in Self-Supervised Learning
Authors: Xi Weng, Yunhao Ni, Tengwei Song, Jie Luo, Rao Muhammad Anwer, Salman Khan, Fahad Khan, Lei Huang
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments on Image Net classification and COCO object detection demonstrate INTL s potential in learning superior representations. |
| Researcher Affiliation | Academia | 1SKLCCSE, Institute of Artificial Intelligence, Beihang University, Beijing, China 2Mohamed bin Zayed University of Artificial Intelligence, UAE |
| Pseudocode | Yes | We provide the full Py Torch-style algorithm in Appendix C... Figure 4: Algorithm of INTL, Py Torch-style Pseudocode. |
| Open Source Code | Yes | The code is available at https://github.com/winci-ai/INTL. |
| Open Datasets | Yes | CIFAR-10 and CIFAR-100 (Krizhevsky, 2009), two small-scale datasets... Image Net-100 (Tian et al., 2020a), a random 100-class subset of Image Net (Deng et al., 2009). Image Net (Deng et al., 2009), the well-known largescale dataset... COCO2017 (Lin et al., 2014), a large-scale object detection, segmentation, and captioning dataset... |
| Dataset Splits | No | The paper mentions 'standard linear evaluation protocol on Image Net' and 'same evaluation protocol as in W-MSE (Ermolov et al., 2021)' for CIFAR-10/100, but does not explicitly provide the training/validation/test split percentages or sample counts for these datasets within the paper. |
| Hardware Specification | Yes | We report time and GPU memory requirements based on our implementation for different settings on Image Net with Res Net-50. The batch size is 256, and we train each model with 2 A100-PCIE-40GB GPUs, using mixed precision and py-torch optimized version of synchronized batch-normalization layers. |
| Software Dependencies | No | The paper mentions 'Py Torch-style pseudocode' and uses 'Adam optimizer (Kingma & Ba, 2014)' but does not provide specific version numbers for PyTorch or any other software dependencies. |
| Experiment Setup | Yes | Optimizer and Learning Rate Schedule. We apply the SGD optimizer, using a learning rate of base-lr Batch Size / 256 and cosine decay schedule. The base-lr for 100-epoch pre-training is 0.5, for 200(400)-epoch is 0.4 and for 800-epoch is 0.3. The weight decay is 10 5 and the SGD momentum is 0.9. In addition, we use learning rate warm-up for the first 2 epochs of the optimizer. Image Transformation Details. In image transformation, We use the same augmentation parameters as BYOL (Grill et al., 2020). Each input image is transformed twice to produce the two distorted views. The image augmentation pipeline consists of the following transformations: random cropping, resizing to 224 224, horizontal flipping, color jittering, converting to grayscale, Gaussian blurring, and solarization. The details of parameters are shown in Table 4. |