Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Hierarchical Poset Decoding for Compositional Generalization in Language
Authors: Yinuo Guo, Zeqi Lin, Jian-Guang Lou, Dongmei Zhang
NeurIPS 2020 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate our proposed decoder on Compositional Freebase Questions (CFQ), a large and realistic natural language question answering dataset that is specifically designed to measure compositional generalization. Results show that it outperforms current decoders. (Abstract) |
| Researcher Affiliation | Collaboration | 1Key Laboratory of Computational Linguistics School of EECS, Peking University, Beijing, China; 2Microsoft Research Asia, Beijing, China |
| Pseudocode | Yes | Algorithm 1 generate_poset (the decoding process at inference time) |
| Open Source Code | No | The paper references external tools with links (GIZA++2, Match-Zoo3) but does not provide a link or explicit statement for the open-source code of their proposed method. |
| Open Datasets | Yes | Dataset We conduct experiments on three different splits MCD1/MCD2/MCD3 of CFQ dataset (Keysers et al., 2020). |
| Dataset Splits | Yes | Each split contains 95k/12k/12k instances for training/development/test. (Section 6.1) |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory, or cloud computing specifications) used for running the experiments. |
| Software Dependencies | No | The paper mentions software such as GIZA++2, Match-Zoo3, and PyTorch, but does not provide specific version numbers for these software dependencies. |
| Experiment Setup | Yes | The dimensions of the hidden state and word embedding are set to 512, 300 respectively. The training process lasts 50 epochs with batch size 64/256 for sketch prediction and traversal path prediction, respectively. We use the Adam optimizer with default settings (in PYTORCH) and a dropout layer with the rate of 0.5. (Section 6.1) |