Unconditional stability of a recurrent neural circuit implementing divisive normalization
Authors: Shivang Rawat, David Heeger, Stefano Martiniani
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | By evaluating the model s performance on RNN benchmarks, we find that ORGa NICs outperform alternative neurodynamical models on static image classification tasks and perform comparably to LSTMs on sequential tasks. and We provide further empirical evidence in support of Conjecture 5.3 that ORGa NICs is asymptotically stable by showing that stability is preserved when training ORGa NICs using na ıve BPTT on two different tasks: 1) static classification of MNIST, 2) sequential classification of pixel-by-pixel MNIST. |
| Researcher Affiliation | Academia | 1 Courant Institute of Mathematical Sciences, NYU 2 Center for Soft Matter Research, Department of Physics, NYU 3 Department of Psychology, NYU 4 Center for Neural Science, NYU 5 Simons Center for Computational Physical Chemistry, Department of Chemistry, NYU |
| Pseudocode | Yes | Algorithm 1 Iterative scheme for the fixed point when the maximum singular value of Wr is 1 |
| Open Source Code | Yes | Python code for this study is available at https://github.com/martiniani-lab/dynamic-divisive-norm. |
| Open Datasets | Yes | We first show that we can train ORGa NICs on the MNIST handwritten digit dataset [72] presented to the circuit as a static input. |
| Dataset Splits | Yes | We performed a random split of 57,000 training samples and 3,000 validation samples and picked the model with the largest validation accuracy for testing. |
| Hardware Specification | Yes | The simulations were performed on an HPC cluster. All of the models were trained on a single A100 (80GB) GPU. |
| Software Dependencies | No | The paper mentions PyTorch ('The code (written in Py Torch [92])') but does not specify a version number or other key software dependencies with their versions. |
| Experiment Setup | Yes | More details about the parameters are given in Table 4; kaiming uniform initialization is used from [94]. Additional hyperparameters are given in Table 7. and Table 7: Hyperparameters Batch size 256 256 Initial Learning rate 0.001 0.01 Weight decay 10 5 10 5 Step size (Step LR) None 30 epochs Gamma (Step LR) None 0.8 |