The Neural Race Reduction: Dynamics of Abstraction in Gated Networks
Authors: Andrew Saxe, Shagun Sodhani, Sam Jay Lewallen
ICML 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate our key insights on naturalistic datasets and with relaxed assumptions. |
| Researcher Affiliation | Collaboration | 1Gatsby Computational Neuroscience Unit & Sainsbury Wellcome Centre, UCL 2FAIR, Meta AI 3CIFAR Azrieli Global Scholar, CIFAR. |
| Pseudocode | No | The paper describes mathematical derivations and computational processes, but it does not include any explicitly labeled 'Pseudocode' or 'Algorithm' blocks. |
| Open Source Code | Yes | The code and results are available at https: //www.saxelab.org/gated-dln. |
| Open Datasets | Yes | We use the MNIST (Deng, 2012) and CIFAR-10 datasets (Krizhevsky et al., 2009) as base datasets, and rotations and permutations as transformations. |
| Dataset Splits | Yes | Following the setup in Section 4, we train a subset of input-output domains such that each input domain is trained with only K M output domains, resulting in M K trained pathways and M (M K) untrained pathways. During evaluation, we report the performance on both the trained pathways and the untrained pathways. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU/CPU models, memory) used to run the experiments. It only refers to 'simulated networks'. |
| Software Dependencies | No | The paper lists libraries such as Py Torch (Paszke et al., 2019), Hydra (Yadan, 2019), and Numpy (Harris et al., 2020), but it does not provide explicit version numbers for these software dependencies (e.g., 'PyTorch 1.9' instead of just the citation). |
| Experiment Setup | Yes | Table 1 lists 'Hyperparameter values common across all the task distributions', including 'Batch size (per task) 8', 'Number of epochs 1000', 'Learning Rate 0.0001', and 'Momentum 0.9'. |