On Sparse Modern Hopfield Model
Authors: Jerry Yao-Chieh Hu, Donglin Yang, Dennis Wu, Chenwei Xu, Bo-Yu Chen, Han Liu
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, we use both synthetic and real-world datasets to demonstrate that the sparse Hopfield model outperforms its dense counterpart in many situations. |
| Researcher Affiliation | Academia | Department of Computer Science, Northwestern University, Evanston, IL 60208 USA Department of Physics, National Taiwan University, Taipei 10617, Taiwan Department of Statistics and Data Science, Northwestern University, Evanston, IL 60208 USA |
| Pseudocode | Yes | See (D.12) and the implementation Algorithm 1 in Appendix D, and [Ramsauer et al., 2021, Section 3] for more details of these associations. |
| Open Source Code | Yes | Code is available at Git Hub; future updates are on arXiv. [Version: Novermber, 28, 2023] |
| Open Datasets | Yes | For the memory capacity (the top row of Figure 1), we test the proposed sparse model on retrieving half-masked patterns comparing with the Dense (Softmax) and 10th order polynomial Hopfield models [Millidge et al., 2022, Krotov and Hopfield, 2016] on MNIST (high sparsity), Cifar10 (low sparsity) and Image Net (low sparsity) datasets. ... Next, we demonstrate that the proposed method achieves near-optimal performance on four realistic (non-sparse) MIL benchmark datasets: Elephant, Fox and Tiger for image annotation [Ilse et al., 2018], UCSB breast cancer classification [Kandemir et al., 2014]. |
| Dataset Splits | Yes | The training loss and accuracy curve of dense and sparse Hopfield models with different bag sizes. Bottom: The validation loss and accuracy curve of dense and sparse Hopfield models with different bag sizes. The plotted are the mean of 10 runs. ... For the memory capacity (the top row of Figure 1), we test the proposed sparse model on retrieving half-masked patterns comparing with the Dense (Softmax) and 10th order polynomial Hopfield models [Millidge et al., 2022, Krotov and Hopfield, 2016] on MNIST (high sparsity), Cifar10 (low sparsity) and Image Net (low sparsity) datasets. |
| Hardware Specification | No | This research was supported in part through the computational resources and staff contributions provided for the Quest high performance computing facility at Northwestern University which is jointly supported by the Office of the Provost, the Office for Research, and Northwestern University Information Technology. This statement mentions a computing facility but does not provide specific hardware details such as GPU models or CPU specifications. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | For all Hopfield models, we set β = 1. A query is regarded as correctly retrieved if its cosine similarity error is below a set threshold. ... For MNIST/CIFAR10/Image Net datasets, we set the error thresholds to be 10/20/20 to cope with different sparse levels in data. ... a detailed description of this experiment as well as its training and evaluating process can be found in Appendix H.1.2. |