Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Alignment of Large Language Models with Constrained Learning
Authors: Botong Zhang, Shuo Li, Ignacio Hounie, Osbert Bastani, Dongsheng Ding, Alejandro Ribeiro
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the effectiveness and merits of our iterative dual-based alignment method through extensive experiments on the PKU-Safe RLHF [23] and Anthropic HH-RLHF [2] datasets. |
| Researcher Affiliation | Collaboration | Botong Zhang University of Pennsylvania EMAIL Shuo Li Amazon EMAIL Ignacio Hounie University of Pennsylvania EMAIL Osbert Bastani University of Pennsylvania EMAIL Dongsheng Ding University of Tennessee, Knoxville EMAIL Alejandro Ribeiro University of Pennsylvania EMAIL |
| Pseudocode | Yes | Algorithm 1 Constrained Alignment via Iterative Dualization (CAID) |
| Open Source Code | Yes | Code available at: https://github.com/botong516/Constrained-LLMs |
| Open Datasets | Yes | We demonstrate the effectiveness and merits of our iterative dual-based alignment method through extensive experiments conducted on the PKU-Safe RLHF [23] and Anthropic HH-RLHF [2] datasets. |
| Dataset Splits | Yes | In each dual subgradient step, we sample 600 prompts from the training split and generate 64 responses using the updated model from the previous iteration to compute the subgradient direction. To make a fair comparison, we set the total number of iterations T to be the number of epochs used in the LLM policy optimization step (i.e., DPO) of the one-shot method; in our method, we perform one epoch per iteration, and initialize the dual variable by an one-shot solution as a practical, zero-cost warm start. |
| Hardware Specification | Yes | We conduct our experiments using five 48GB NVIDIA A6000 GPUs for model updates and three such GPUs for generating and evaluating on-policy responses to update the dual variable. |
| Software Dependencies | No | The paper does not explicitly state software dependencies with specific version numbers (e.g., Python, PyTorch, CUDA versions). |
| Experiment Setup | Yes | Table 1 and 2 report the DPO training hyperparameters for the single-constraint and multi-constraint settings, respectively. Table 3 resports the configuration used for model generation. |