CoNT: Contrastive Neural Text Generation
Authors: Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, Xuanjing Huang
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate CONT on five generation tasks with ten benchmarks, including machine translation, summarization, code comment generation, datato-text generation and commonsense generation. Experimental results show that CONT clearly outperforms the conventional training framework on all the ten benchmarks with a convincing margin. |
| Researcher Affiliation | Collaboration | Chenxin An1,2 , Jiangtao Feng2, Kai Lv1, Lingpeng Kong2,3, Xipeng Qiu1, Xuanjing Huang1,4 1Fudan University, 2Shark-NLP Shanghai AI Laboratory 3The University of Hong Kong 4Shanghai Collaborative Innovation Center of Intelligent Visual Computing |
| Pseudocode | Yes | Algorithm 1 Inference algorithm: Given an input sequence x, a contrastive generation model M = ( f, g); return the output sequence. Full details of the training algorithm can be found in Algorithm 2, Appendix B. |
| Open Source Code | Yes | 2The code is available at https://github.com/Shark-NLP/Co NT |
| Open Datasets | Yes | We validate CONT on five generation tasks with ten benchmarks, including machine translation, summarization, code comment generation, datato-text generation and commonsense generation. [28], [9], [27], [22], [33], [24]. |
| Dataset Splits | Yes | In most cases, can be directly set to 0.5, tuning on the validation set will usually get better results. The test set and dev set of Totto are both split into two parts overlap and non-overlap. Details of our experimental setup for each benchmarks can be found in Appendix C. |
| Hardware Specification | No | The paper does not provide specific details on the hardware used for running experiments (e.g., GPU models, CPU types, or cloud computing instance specifications). |
| Software Dependencies | No | The paper mentions general tools like 'fairseq' and 'Huggingface’s transformers' in its references, but does not provide specific version numbers for any software dependencies (e.g., Python 3.x, PyTorch 1.x, CUDA x.x) used in the experiments. |
| Experiment Setup | Yes | In most cases, can be directly set to 0.5, tuning on the validation set will usually get better results. A warm-up stage where the model is only supervised by LNLL is recommended as it guarantees the quality of the examples from the model s prediction. Details of our experimental setup for each benchmarks can be found in Appendix C. |