Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Avoiding exp(R) scaling in RLHF through Preference-based Exploration
Authors: Mingyu Chen, Yiding Chen, Wen Sun, Xuezhou Zhang
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, our systematic evaluation confirms that SE-POPO is more sample-efficient than both exploratory and non-exploratory baselines, in two primary application scenarios of RLHF as well as on public benchmarks, marking a significant step forward in RLHF algorithm design. |
| Researcher Affiliation | Academia | Mingyu Chen Department of Electrical & Computer Engineering Boston University Boston, MA 02215 EMAIL Yiding Chen Department of Computer Science Cornell University Ithaca, NY 14850 EMAIL Wen Sun Department of Computer Science Cornell University Ithaca, NY 14850 EMAIL Xuezhou Zhang Faculty of Computing & Data Sciences Boston University Boston, MA 02215 EMAIL |
| Pseudocode | Yes | Algorithm 1 SE-POPO: Self-Exploring Preference-Incentive Online Preference Optimization Algorithm 2 POPO: Preference-Incentive Online Preference Optimization |
| Open Source Code | Yes | We provide an anonymized version of data and code as supplemental materials. |
| Open Datasets | Yes | RLHFlow-ultrafeedback dataset as the training prompt sets, and GRM-Llama3-8B-rewardmodel-ft as the training preference model. More details about the experiment setup are deferred to Appendix K.1. The results from the three sets of experiments are shown as three columns in Table 1: 5https://huggingface.co/datasets/RLHFlow/ultrafeedback_iter1, https://huggingface.co/datasets/RLHFlow/ultrafeedback_iter2, https://huggingface.co/datasets/RLHFlow/ultrafeedback_iter3 6https://huggingface.co/Ray2333/GRM-Llama3-8B-rewardmodel-ft |
| Dataset Splits | No | IID data" refers to the setting where the models are evaluated on a held-out test prompt set that are drawn from the same distribution as the training prompt set, and the responses are evaluated by the same preference model used during training. This is to simulate Use Case #1. Alpaca data" refers to the setting where the models are evaluated on the Alpaca Eval 2.0 dataset, but the responses are still evaluated by the same preference model used during training. This is to simulate Use Case #2. |
| Hardware Specification | Yes | The experiments were conducted on 4 x Nvidia A100 80G GPUs. |
| Software Dependencies | No | For all experiments, our implementation build upon the iterative DPO codebase from [Dong et al., 2024], and we use the 3-iteration online RLHF framework following the setting in [Xie et al., 2024]. Across all three experiments, we use Llama-3-8B-SFT as the base model 2, RLHFlow-ultrafeedback dataset as the training prompt sets, and GRM-Llama3-8B-rewardmodel-ft as the training preference model. |
| Experiment Setup | Yes | For hyperparameters, we mainly follow the settings in Xie et al. [2024] and Zhang et al. [2024]. We set β = 0.1, use a global batch size of 128, use a learning rate of 5 10 7 with cosine scheduling. For exploration coefficient α, we employ a decreasing strategy across iterations as in Xie et al. [2024] and do a grid search for α in the first iteration over {0.1, 0.01, 0.001, 0.0001, 0.00001}. Based on the empirical performance on Alphca Eval benchmark, we finally select {1 10 3, 5 10 4, 0} for XPO and {1 10 1, 5 10 2, 0} for SE-POPO respectively. |