Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Attractive or Faithful? Popularity-Reinforced Learning for Inspired Headline Generation
Authors: Yun-Zhu Song, Hong-Han Shuai, Sung-Lin Yeh, Yi-Lun Wu, Lun-Wei Ku, Wen-Chih Peng8910-8917
AAAI 2020 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through quantitative and qualitative experiments, we show that the proposed PORL-HG significantly outperforms the state-of-the-art headline generation models in terms of attractiveness evaluated by both human (71.03%) and the predictor (at least 27.60%), while the faithfulness of PORL-HG is also comparable to the state-of-the-art generation model. |
| Researcher Affiliation | Academia | 1National Chiao Tung University, Taiwan, 2National Tsing Hua University, Taiwan 3Academia Sinica, Taiwan |
| Pseudocode | No | The paper describes its methods in detail through narrative text and diagrams (e.g., Figure 1 and Figure 2 illustrate the framework and extractor architecture) but does not include any explicit pseudocode blocks or algorithms labeled as such. |
| Open Source Code | No | The paper states 'Moreover, the datasets will be released as a public download for future research.' and provides a link: 'The details of the analysis and datasets are available at https://github.com/yunzhusong/AAAI20-PORLHG.' This explicitly refers to datasets, not the source code for the proposed methodology, and a check of the provided link shows an empty repository. |
| Open Datasets | Yes | Therefore, we build the datasets CNNDM-DH (CNN/Daily Mail-Document with Headline) and DM-DHC (Daily Mail Document with Headline and Comment) based on CNN/Daily Mail dataset (Nallapati et al. 2016; Hermann et al. 2015)... The details of the analysis and datasets are available at https://github.com/yunzhusong/AAAI20-PORLHG. |
| Dataset Splits | Yes | Table 1: Dataset information Train Val Test CNNDM-DH 281208 12727 10577 DM-DHC 138787 11862 10130 |
| Hardware Specification | No | The paper does not specify the hardware used for running experiments, such as particular GPU or CPU models, memory specifications, or cloud computing instance types. |
| Software Dependencies | No | The paper discusses software components like CNN, LSTM, and word2vec (e.g., 'we exploit a convolutional neural network (Kim 2014)', 'a bidirectional LSTM is then applied', 'we first embed each word by word2vec'), but it does not provide specific version numbers for any programming languages, libraries, or frameworks used for implementation. |
| Experiment Setup | No | The paper describes the model architecture and training procedures (e.g., 'We make the sentence extractor into an RL agent', 'pre-train the abstractor, extractor and classi๏ฌer'), but it does not explicitly state specific hyperparameter values such as learning rates, batch sizes, number of epochs, or optimizer settings necessary to reproduce the experimental setup. |