Alignment with human representations supports robust few-shot learning
Authors: Ilia Sucholutsky, Tom Griffiths
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We confirm this prediction empirically, finding such a relationship in an analysis of the performance of 491 computer vision models. We also show that highly-aligned models are more robust to both natural adversarial attacks and domain shifts. Our results suggest that human alignment is often a sufficient, but not necessary, condition for models to make effective use of limited data, be robust, and generalize well. |
| Researcher Affiliation | Academia | Ilia Sucholutsky Department of Computer Science Princeton University is2961@princeton.edu Thomas L. Griffiths Departments of Psychology and Computer Science Princeton University tomg@princeton.edu |
| Pseudocode | No | The paper contains mathematical definitions and proofs (Lemma 3.7, Proposition 3.8, Theorem 3.9, etc.) and discusses models and experiments, but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | All code and full results data are provided as part of the supplemental information. We will share them publicly after the anonymity period is over. |
| Open Datasets | Yes | All of the models used in this paper were pre-trained on Image Net-1k [31] and had their performance evaluated on the Image Net-1k validation set. Few-shot transfer learning performance was evaluated on the CIFAR100 [18] test set with n {1, 5, 10, 20, 40, 80} examples per class used for few-shot learning and the remaining examples used for evaluation. We measure alignment by computing the three metrics described below on the six image datasets from Peterson et al. [29]... We now test few-shot transfer learning performance on the MNIST [21] and FMNIST [42] datasets |
| Dataset Splits | No | All of the models used in this paper were pre-trained on Image Net-1k [31] and had their performance evaluated on the Image Net-1k validation set. Few-shot transfer learning performance was evaluated on the CIFAR100 [18] test set with n {1, 5, 10, 20, 40, 80} examples per class used for few-shot learning and the remaining examples used for evaluation. |
| Hardware Specification | Yes | All experiments were conducted on an AWS x1.16xlarge instance (no GPUs). |
| Software Dependencies | No | For linear probing, we take the embeddings coming from the penultimate layer of a model and fit a logistic regression using scikit-learn [28]. For the classifier heads, we use a oneand two-hidden layer neural network implemented in Py Torch [27]... compute partial correlation with Image Net-1k Top-1 validation accuracy as a covariate using the Pingouin Python package [39]. |
| Experiment Setup | Yes | For the classifier heads, we use a oneand two-hidden layer neural network implemented in Py Torch [27], both with an input dropout rate of 0.8 and the latter with a Re LU hidden layer. The heads are provided in the supplement and fixed hyperparameter values (selected based on exploratory experiments to confirm stable convergence) were used for every dataset/model for consistency. |