Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Computational Budget Should Be Considered in Data Selection
Authors: Weilin Wan, Weizhong Zhang, Cheng Jin
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments show that our method achieves performance gains of up to 14.42% over baselines in vision and language benchmarks. Additionally, CADS achieves a 3-20 speedup compared to conventional bilevel implementations, with acceleration correlating positively with compute budget size. |
| Researcher Affiliation | Academia | Weilin Wan1, Weizhong Zhang2 , Cheng Jin1 1College of Computer Science and Artificial Intelligence, Fudan University 2School of Data Science, Fudan University EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1 CADS-E (example-level) Algorithm 2 CADS-S (source-level) |
| Open Source Code | Yes | Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: we provide detailed instructions for accessing the data and code. |
| Open Datasets | Yes | We validate CADS on a reduced-scale MNIST classification task... We evaluate CADS-S on a grouped CIFAR-10 variant... We evaluate CADS-S on Domain Net... We evaluated our CADS-S approach on instruction fine-tuning tasks using the GPT-2 model [48]. The dataset consists of two heterogeneous data sources: (i) Alpaca GPT4 [46]... (ii) Alpaca [57]... We extended our evaluation to 13 distinct instruction-following corpora... These included the original Alpaca GPT4 benchmark[46], the Slim Orca set[32], the Alpaca collection[57], the GPTeacher suite[58], and nine multilingual variants of the Alpaca data. |
| Dataset Splits | Yes | All experiments use a fixed training set of 1,000 examples and batch size 1,000. We optimize under an epoch-based budget of 20 epochs... A fixed validation set of 10,000 points is used to track generalization error... Specifically, we reserved 10% of the training set (5,000 samples) as a validation set... The remaining examples are split into training, validation, and test subsets according to either explicitly provided sizes or a default 90/5/5 ratio. |
| Hardware Specification | Yes | All experiments were run on a single machine equipped with an NVIDIA A100 80 GB GPU under CUDA 12.6 and NVIDIA driver 470.199.02, using PyTorch 2.5.1. |
| Software Dependencies | Yes | All experiments were run on a single machine equipped with an NVIDIA A100 80 GB GPU under CUDA 12.6 and NVIDIA driver 470.199.02, using PyTorch 2.5.1... On the software side, we build atop PyTorch and torchvision (BSD 3-Clause), SciPy (BSD), and Hugging Face Transformers (GPT-2, Apache 2.0). |
| Experiment Setup | Yes | We train for 160 steps using Adam (learning rate 3e-4) and batch size 1000... We solve the bilevel problem with Adam for both the network parameters θ (learning rate 5e-3) and the selection weights s (learning rate 5e-2). The outer loop runs for 300 iterations with variance reduction and gradient clipping enabled... We allocate an epoch-based compute budget of 2 to 5 epochs... We adopt the standard ResNet-18 backbone for all runs. |