GULP: a prediction-based metric between representations
Authors: Enric Boix-Adsera, Hannah Lawrence, George Stepaniants, Philippe Rigollet
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In Section 4 we validate GULP through extensive experiments3. Finally, we conclude in Section 5. |
| Researcher Affiliation | Academia | Enric Boix-Adserà MIT eboix@mit.edu Hannah Lawrence MIT hanlaw@mit.edu George Stepaniants MIT gstepan@mit.edu Philippe Rigollet MIT rigollet@math.mit.edu |
| Pseudocode | No | No structured pseudocode or algorithm blocks were found in the paper. |
| Open Source Code | Yes | Our code is available at https://github.com/sgstepaniants/GULP. |
| Open Datasets | Yes | Figure 1: t-SNE embedding of various pretrained DNN representations of the Image Net [KSH12] dataset with GULP distance (λ = 10 2) and trained on the MNIST handwritten digit database. and independently train 16 Resnet18 architectures on the CIFAR10 dataset [KH+09] |
| Dataset Splits | Yes | We then draw n = 5,000 images from the dataset X1, . . . , Xn PX, and assign a random label Yk N(0, 1) to each one. For each representation i 2 [m], we fit a λ-regularized least-squares linear regression to the training data {(Xk, Yk)}k2[n]... by taking the empirical average over 3000 samples in a test set. |
| Hardware Specification | No | No specific hardware details (exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running its experiments were provided. |
| Software Dependencies | No | No specific ancillary software details (e.g., library or solver names with version numbers) needed to replicate the experiment were provided. |
| Experiment Setup | Yes | We consider the representation maps φ1, . . . , φm given by m = 37 pretrained image classification architectures on the Image Net dataset PX (see Appendix B.5). For each pair of representations, we estimate the CKA, CCA, PWCCA, and GULP distances, using the plug-in estimators on 10,000 images, sufficient to guarantee good convergence (see Figure 3). We then draw n = 5,000 images from the dataset X1, . . . , Xn PX, and assign a random label Yk N(0, 1) to each one. For each representation i 2 [m], we fit a λ-regularized least-squares linear regression to the training data {(Xk, Yk)}k2[n], which gives a coefficient vector βλ,i. We input all MNIST training set images into each network, save their representations at the final hidden layer, and compute CKA, PROCRUSTES, and GULP distances between all pairs of representations. independently train 16 Resnet18 architectures on the CIFAR10 dataset [KH+09] for 50 epochs. |