Better than Random: Reliable NLG Human Evaluation with Constrained Active Sampling
Authors: Jie Ruan, Xiao Pu, Mingqi Gao, Xiaojun Wan, Yuesheng Zhu
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiment results on 137 real NLG evaluation setups with 44 human evaluation metrics across 16 datasets and 5 NLG tasks demonstrate CASF receives 93.18% top-ranked system recognition accuracy and ranks first or ranks second on 90.91% of the human metrics with 0.83 overall inter-system ranking Kendall correlation. |
| Researcher Affiliation | Academia | Peking University {ruanjie,puxiao}@stu.pku.edu.cn, {gaomingqi,wanxiaojun,zhuys}@pku.edu.cn |
| Pseudocode | No | The paper describes methods and rules in prose and diagrams (e.g., Figure 3), but it does not include explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code and data are publicly available online.1 ... 1https://github.com/Enabler Rx/CASF |
| Open Datasets | Yes | The datasets are: Summarization (SUM): We utilize 8 human evaluation datasets of the model generated summarization, which are Summ Eval (2021), REALSumm (2020), Newsroom (Ne R18) (2018), Dial Summ Eval (Dial Summ) (2022) and Open AIaxis1 (Open AI 1) (2020; 2017), Open AI-axis2 (Open AI 2) , Open AI-CNN/DM1 (Open AI 3) , and Open AI-CNN/DM3 (Open AI 4) . Machine Translation (MT): We use 3 datasets collected from WMT news translation tasks (2021) viz. newstest2020 en-de (newstest 1), newstest2020 cn-en (newstest 2) and newstest2021 cn-en (newstest 3). |
| Dataset Splits | Yes | Details of the datasets, preprocessing and the validation set for hyper-parameters selection are in the Tasks and Dataset section of Appendix. |
| Hardware Specification | No | The paper does not explicitly describe the specific hardware (e.g., CPU, GPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper mentions using Gradient Boosting Decision Tree (GBDT) and various automatic metrics (BERT-SCORE, MOVER-SCORE, ROUGE, BART-SCORE, BLEU, METEOR) but does not provide specific version numbers for any software or libraries. |
| Experiment Setup | Yes | We compare methods with 50% sampling rate. Results for other sampling ratios are in Different Sampling Ratio section of Appendix. In addition, the number of phases and the sampling ratio of each phase are 5 and 10%. The determination of these parameters is shown in the Phases and Associated Sampling Ratios section. |