Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Grammar-Aligned Decoding
Authors: Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick, Nadia Polikarpova, Loris D'Antoni
NeurIPS 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our evaluation on code generation and structured NLP tasks shows how ASAp often produces outputs with higher likelihood (according to the LLM s distribution) than existing GCD techniques, while still enforcing the desired grammatical constraints. We evaluate ASAp on two structured prediction tasks: formal program synthesis and constituency parsing. |
| Researcher Affiliation | Academia | University of Wisconsin-Madison University of California San Diego |
| Pseudocode | Yes | Algorithm 1 ASAp algorithm |
| Open Source Code | Yes | Our code, datasets, and checkpoints are available at: https://github.com/ebmoon/transformers-GAD. |
| Open Datasets | Yes | Our code, datasets, and checkpoints are available at: https://github.com/ebmoon/transformers-GAD. |
| Dataset Splits | Yes | We employ a standard train-validation-test split of 70-10-20%. |
| Hardware Specification | Yes | Our experiments are conducted on 4 NVIDIA RTX A6000 GPUs and 4 NVIDIA A100 GPUs. |
| Software Dependencies | Yes | Our implementation is based on Python 3.10 and Py Torch 2.1.2. |
| Experiment Setup | Yes | Key hyperparameters include a learning rate of 2e-4, a warmup ratio of 0.03, a maximum sequence length of 2048, Lo RA alpha of 32, Lo RA dropout of 0.05, and Lo RA rank of 64. |