RePre: Improving Self-Supervised Vision Transformer with Reconstructive Pre-training
Authors: Luya Wang, Feng Liang, Yangguang Li, Honggang Zhang, Wanli Ouyang, Jing Shao
IJCAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments demonstrate the effectiveness and portability of this method. We validate our Re Pre in the latest contrastive learning frameworks (e.g., DINO, MOCO V3, Mo BY, BYOL and Sim CLR). Following standard linear evaluation on Image Net-1K, with Re Pre, these methods improve top-1 accuracy by 0.5 1.1%. Prominently, it also brings significant performance to the base methods on dense prediction tasks on the COCO and cityscape datasets, even outperforming supervised methods. |
| Researcher Affiliation | Collaboration | Luya Wang1 , Feng Liang2 , Yangguang Li3 , Honggang Zhang1 , Wanli Ouyang4 , Jing Shao3 1Beijing University of Posts and Telecommunications 2University of Texas at Austin 3Sense Time Group Limited 4The University of Sydney |
| Pseudocode | Yes | Algorithm 1 Pseudo code of Re Pre in a Py Torch-like style |
| Open Source Code | No | The paper does not contain an explicit statement about releasing the source code or provide a link to a code repository. |
| Open Datasets | Yes | Following standard linear evaluation on Image Net-1K, with Re Pre, these methods improve top-1 accuracy by 0.5 1.1%. We further evaluate the transferring performance of the learned representations on downstream tasks of COCO object detection/instance segmentation and Cityscapes semantic segmentation. |
| Dataset Splits | No | The paper mentions evaluating on ImageNet-1K and other datasets, implying the use of standard splits. However, it does not provide specific details on the training, validation, or test dataset splits (e.g., exact percentages or sample counts), or mention cross-validation setups. |
| Hardware Specification | Yes | All our experiments are conducted on NVIDIA V100 GPUs. |
| Software Dependencies | No | The paper mentions 'Py Torch-like style' in its pseudocode description, but it does not specify version numbers for PyTorch or any other software dependencies, such as CUDA, specific Python versions, or other libraries. |
| Experiment Setup | Yes | Following standard practice, we use Adam W optimizer and 1 schedule. The shorter edges of the input images are resized to 800 while the longer side is at most 1333. |