Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations
Authors: Yongyi Yang, Jacob Steinhardt, Wei Hu
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To study the above question, we design experiments to manually create a mismatch between the intrinsic structure of the input distribution and the explicit labels provided for training in standard classification datasets and measure how the last-layer representations behave in response to our interventions. |
| Researcher Affiliation | Academia | 1University of Michigan, Ann Arbor, Michigan, USA 2UC Berkeley, Berkeley, California, USA. |
| Pseudocode | No | The paper includes mathematical proofs and derivations, but does not present any pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any links to open-source code for the methodology described. |
| Open Datasets | Yes | As an example, if we train on CIFAR-10 using only 5 coarse-grained labels (by combining two classes into one superclass) until convergence, we can reconstruct the original 10-class labels from the learned representations via unsupervised clustering. |
| Dataset Splits | No | The paper mentions using CIFAR-10 and its test set but does not explicitly describe the training/validation/test dataset splits or proportions in the main text. |
| Hardware Specification | No | The paper does not specify the hardware (e.g., GPU, CPU models) used for running the experiments. |
| Software Dependencies | No | The paper does not list specific software dependencies with version numbers (e.g., Python, PyTorch versions). |
| Experiment Setup | Yes | Specifically, we choose the initial learning rate in {10 1, 10 2, 10 3} (we apply a standard learning rate decay schedule) and the weight-decay rate in {5 10 3, 5 10 4, 5 10 5} and run all 9 possible combinations of them. |