Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Memory Mosaics at scale
Authors: Jianyu Zhang, Leon Bottou
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Throughout the evaluation, memory mosaics v2 match transformers on the learning of training knowledge (first dimension) and significantly outperforms transformers on carrying out new tasks at inference time (second and third dimensions). These improvements cannot be easily replicated by simply increasing the training data for transformers. A memory mosaics v2 trained on one trillion tokens still perform better on these tasks than a transformer trained on eight trillion tokens. |
| Researcher Affiliation | Collaboration | Jianyu Zhang New York University, New York FAIR, Meta Inc., New York Lรฉon Bottou FAIR, Meta Inc., New York New York University, New York |
| Pseudocode | No | The paper describes the methodology using mathematical equations and descriptive text, but does not include any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | 2https://github.com/facebookresearch/Memory Mosaics |
| Open Datasets | Yes | Tasks description To assess the in-context learning ability, we employ classic multiclass classification problems,12 adopted from Li et al. [2024]. The classification tasks include: Banking77 [Casanueva et al., 2020] is a banking-intent classification task with 77 target categories. Tacred [Zhang et al., 2017] is a relation classification task of two objects in a sentence, extracted from newswire or webtext, with 41 target categories. Goemotion [Demszky et al., 2020] is an emotion classification task of Reddit comments with 28 target categories. |
| Dataset Splits | No | The paper describes training on "200 billion tokens of a diverse datamix" and "1 trillion tokens of the same datamix" but does not explicitly provide specific training/validation/test splits, percentages, or sample counts for these datasets in the context of model training. |
| Hardware Specification | Yes | All experiments are conducted on H100 GPUs with 80GB VRAM. |
| Software Dependencies | No | The paper mentions software components like "adamw optimizer" and uses "llama architecture", but does not provide specific version numbers for any software libraries or dependencies, such as Python, PyTorch, or CUDA versions. |
| Experiment Setup | Yes | For all Memory Mosaics v2 and baseline Transformer models16, we use a consistent set of hyperparameters. That is, a batch size of 1024, a sequence length of 4096, an adamw optimizer with ฮฒ1 = 0.9 and ฮฒ2 = 0.95 accompanied by a L2 weight decay of 0.1 and a gradient norm clip of 1, a learning rate warm-up of 2000 iterations followed by a cosine learning rate scheduler that reduces the learning rate by a factor of 100 at the end. The initial learning rates (after warm-up) are set to 3e-4 for small models and 1e-3 for large models. |