Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Language Modeling by Language Models
Authors: Junyan Cheng, Peter Clark, Kyle Richardson
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We report experiments involving 1,162 newly discovered designs (1,062 fully verified through pre-training) and find the best designs to be highly competitive with known architectures (e.g., outperform GPT2, Mamba2, etc., on 6/9 common benchmarks). We empirically test the core components and overall effectiveness of our Genesys system. |
| Researcher Affiliation | Collaboration | Junyan Chengα Peter Clarkβ Kyle Richardsonβ Allen Institute for AIβ Dartmouth Collegeα EMAIL EMAIL |
| Pseudocode | Yes | In this section, we provide the extended algorithmic details of our designers and different components of Genesys. Algorithm 1 The Design Process (DESIGNMODEL) [...] Algorithm 6 Genesys Evolutionary Loop |
| Open Source Code | Yes | All code and discovery artifacts (e.g., new designs, agent interactions and dialogues) can be found at https://genesys.allen.ai (live console) and https://github.com/allenai/genesys (system code). |
| Open Datasets | Yes | We build our corpus upon Smol LM Allal et al. (2024), a high-quality dataset for training high-performance small LMs. Firstly, we filtered samples with score >= 4 from Fine Web-edu Penedo et al. (2024)... All other subsets (Cosmopedia-v2, Open Web Math Paster et al. (2024), Deep Mind Math Saxton et al. (2019), and Stack Overflow 9) do not provide the sample rating; thus, we randomly select from them. |
| Dataset Splits | Yes | This results in an around 1/8 high-quality subset of Smol LM Corpus, which we call Smol LM-1/8-Corpus. Statistics presented in Table 9. Following Gao et al. (2020), we randomly sampled 1GB of data from the original Smol LM for each of the test and eval sets, respectively, then removed any verbatim from the training set. |
| Hardware Specification | Yes | Our experiments are carried out mainly in a set of 10 machines from our internal cluster. There are 8 machines used as the V-Nodes, including three machines with 8 Nvidia A6000 48GB v RAM GPUS with 124 Cloud CPUs and 512G RAM, and five machines with 8 Nvidia L40S 48GB v RAM GPUS with 256 Cloud CPUs and 1TB RAM. Two machines with 3 Nvidia A6000 48GB v RAM GPUS with 34 Cloud CPUs and 254.3 GB RAM are configured as D-Nodes. All machines are running on Ubuntu 20.04. |
| Software Dependencies | No | The paper mentions several software components like PyTorch, LangChain, Huggingface trainer, LM-Eval framework, but does not provide specific version numbers for them. It mentions 'Ubuntu 20.04' which is an OS, but not programming libraries or frameworks with versions. |
| Experiment Setup | Yes | Table 12: Detailed settings for verification engine lists: Context Len. 2045, Optimizer Adam W, Tokenizer Llama-2-7b-hf, LR Sheduler Cosine with min lr, Min lr rate 0.1, Warmup ratio 0.02, Batch size 0.5M tokens, Learning rate (14M 1e-3, 31M 1e-3, 70M 1e-3, 125M 6e-4). |