Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Group-Level Data Selection for Efficient Pretraining
Authors: Zichun Yu, Fei Peng, Jie Lei, Arnold Overwijk, Scott Yih, Chenyan Xiong
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on DCLM 400M-4x, 1B-1x, and 3B-1x show that Group-MATES achieves 3.5%-9.4% relative performance gains over random selection across 22 downstream tasks, nearly doubling the improvements achieved by state-of-the-art individual data selection baselines. |
| Researcher Affiliation | Collaboration | 1Language Technologies Institute, Carnegie Mellon University 2Meta |
| Pseudocode | No | The paper describes methods through mathematical formulations (e.g., Eq. 12) and textual descriptions of sequential processes, but it does not contain a clearly labeled 'Pseudocode' or 'Algorithm' block. |
| Open Source Code | Yes | Our code is open-sourced at https://github.com/facebookresearch/Group-MATES. |
| Open Datasets | Yes | We conduct our main experiments following standard setups in Data Comp-LM (DCLM) [25]... We also conduct experiments on the C4 dataset in Appendix C.4. |
| Dataset Splits | No | The paper mentions evaluating pretrained models with 22 downstream tasks in either zero-shot or few-shot manners, and using a size-128 subset from FLAN as reference data, and a validation set of 1,000 sampled oracle influences for the data influence model. However, it does not explicitly provide the training/test/validation splits for the main pretraining datasets (DCLM or C4) in terms of percentages or sample counts. |
| Hardware Specification | Yes | Total H100 Hours (Figure 3, Table 5) and All experiments are conducted on 8 GPUs (Appendix B). |
| Software Dependencies | Yes | Our relational data influence model is fine-tuned from bge-base-en-v1.5 [46] |
| Experiment Setup | Yes | Table 4: Training hyperparameters. HYPERPARAMETER 400M-4X 1B-1X 3B-1X RELATIONAL DATA INFLUENCE MODEL STEPS 31403 54923 107610 3086 BATCH SIZE 512 256 256 128 SEQUENCE LENGTH 2048 2048 2048 2048 (512 * 4) MAX LEARNING RATE 3E-3 3E-3 3E-3 5E-5 OPTIMIZER ADAMW ADAMW ADAMW ADAMW SCHEDULER COSINE COSINE COSINE COSINE |