Low Tensor Rank Learning of Neural Dynamics
Authors: Arthur Pellegrino, N Alex Cayco Gajic, Angus Chadwick
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | By fitting RNNs of varying rank to large-scale neural recordings during a motor learning task, we find that the inferred weights are low-tensor-rank and therefore evolve over a fixed low-dimensional subspace throughout the entire course of learning. We next validate the observation of low-tensor-rank learning on an RNN trained to solve the same task. Finally, we present a set of mathematical results bounding the matrix and tensor ranks of gradient descent learning dynamics which show that low-tensor-rank weights emerge naturally in RNNs trained to solve low-dimensional tasks. |
| Researcher Affiliation | Academia | Arthur Pellegrino School of Informatics University of Edinburgh pellegrino.arthur@ed.ac.uk N Alex Cayco-Gajic Département d Etudes Cognitives Ecole Normale Supérieure natasha.cayco.gajic@ens.fr Angus Chadwick School of Informatics University of Edinburgh angus.chadwick@ed.ac.uk |
| Pseudocode | Yes | Algorithm 1 Low tensor rank recurrent neural network fit to data |
| Open Source Code | Yes | Code availability. The ltr RNN implementation can be found at https://github.com/ arthur-pe/Ltr RNN. |
| Open Datasets | No | To test whether neural population dynamics during learning are consistent with a low tensor rank framework, we fit ltr RNNs of varying rank to recordings from the motor and premotor cortex of the macaque during a motor learning task in which the subject must adapt to a force field perturbation in order to reach the target endpoint [16]. We are particularly grateful to Matthew Perich for sharing his data. |
| Dataset Splits | Yes | To compare across models, blocks of consecutive entries in time and trials were held-out for cross validation, and the remaining entries of the data tensor were used to fit the parameters of each ltr RNN model. We hold out for testing blocks of trials and time points (100 ms by 50 trials). Table 1: ...Cross-validation Train blocks 1 10 20 Test blocks 1 5 10... |
| Hardware Specification | Yes | Hardware : desktop with an RTX 3090 Nvidia GPU and i7-12700K Intel CPU. |
| Software Dependencies | No | The paper mentions optimizers like ADAM and SGD, and general components like 'differentiable adaptive step SDE solver', but it does not specify version numbers for any programming languages, libraries, frameworks, or specific solver implementations. |
| Experiment Setup | Yes | Table 1: Hyperparameters of the ltr RNN models. ...When training, we initialize ar N(0, I), br = ar. Furthermore, the weights are parameterized such that ||ar|| = ||br|| = 1... Throughout this work, we used a fully-connected 3-layer DNN with layers of size 150 and Re LU nonlinearities. The dynamical system as a whole is evaluated with a differentiable adaptive step SDE solver [40] and trained with ADAM [53] during initial training, and SGD during motor perturbation learning. |