Scaling Laws for Associative Memories
Authors: Vivien Cabannes, Elvis Dohmatob, Alberto Bietti
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We provide extensive numerical experiments to validate and interpret theoretical results, including fine-grained visualizations of the stored memory associations. |
| Researcher Affiliation | Industry | Vivien Cabannes FAIR, Meta Elvis Dohmatob FAIR, Meta Alberto Bietti Flatiron Institute |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any statement or link indicating that the source code for its methodology is publicly available. |
| Open Datasets | No | The paper states: “The data follows a Zipf law with α = 0.5, N = 100, M = 5 and f(x) = x mod. M.” indicating that synthetic data was generated according to a specified distribution, not obtained from a publicly available dataset with concrete access information. |
| Dataset Splits | No | The paper studies generalization error as a function of the number of data (T) and model capacity (d), rather than providing specific training, validation, or test dataset splits from a fixed dataset. |
| Hardware Specification | No | The paper does not specify any hardware details (e.g., CPU, GPU models, or cloud computing resources) used for running experiments. |
| Software Dependencies | No | The paper does not provide specific version numbers for any software libraries or dependencies used in the experiments. While PyTorch is cited in the bibliography, its version is not specified as a dependency for reproducibility. |
| Experiment Setup | Yes | All the experiments are conducted with N = 100, M = 5, α = 2, f (x) = x mod M, averaged over ten runs. We initialized parameters and rescale learning rates to ensure maximal feature updates, as explained in Appendix B.1. |