Argot: Generating Adversarial Readable Chinese Texts
Authors: Zihan Zhang, Mingxuan Liu, Chao Zhang, Yiming Zhang, Zhou Li, Qi Li, Haixin Duan, Donghong Sun
IJCAI 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this paper, we analyze the differences between Chinese and English, and explore the methodology to transform the existing English adversarial generation method to Chinese. We propose a novel black-box adversarial Chinese texts generation solution Argot, by utilizing the method for adversarial English samples and several novel methods developed on Chinese characteristics. Argot could effectively and efficiently generate adversarial Chinese texts with good readability. Furthermore, Argot could also automatically generate targeted Chinese adversarial text, achieving a high success rate and ensuring readability of the Chinese. (Abstract) ... We have evaluated Argot s performance of success rate, perturbation, time consumption and readability, on several NLP models for both targeted and non-targeted attacks. The results showed that, in average, Argot could generate adversarial texts with a success rate over 97.7% for non-targeted attack, by only introducing less than 11.6% perturbations to original texts, and 98.73% success rate with 12.25% perturbations for targeted attack. (Introduction) ... The entire Section 4 is titled "Evaluation" and discusses experimental setup, datasets, target models, metrics, and presents detailed results in tables and figures. |
| Researcher Affiliation | Collaboration | Zihan Zhang1 , Mingxuan Liu1 , Chao Zhang1,2 , Yiming Zhang1 , Zhou Li3 , Qi Li1,2 , Haixin Duan1,2,4 , Donghong Sun1,2 1Tsinghua University 2Beijing National Research Center for Information Science and Technology 3University of California Irvine 4Qi An Xin Group |
| Pseudocode | Yes | Algorithm 1 : The detail of function glyph(c). |
| Open Source Code | No | The paper references several third-party tools and datasets with GitHub links (e.g., "https://github.com/fxsjy/jieba", "https://github.com/SophonPlus/ChineseNlpCorpus", "https://github.com/kfcd/chaizi", "https://github.com/zzboy/chinese"), but it does not provide a link or an explicit statement about the availability of the source code for their proposed solution, Argot. |
| Open Datasets | Yes | We create a dataset for sentiment classification using samples from an existing Chinese NLP dataset5. (Section 4.1) ... 5https://github.com/SophonPlus/ChineseNlpCorpus (footnote) ... For news classification, we use the THUNews dataset [Sun et al., 2016]. (Section 4.1) |
| Dataset Splits | Yes | Our dataset is composed of 15,471 negative reviews and 16,188 positive reviews. We manually check all these reviews and delete the ones that are too short or have ambiguous labels. We divide this dataset into 80% and 20% for training and validation respectively. For news classification, we use the THUNews dataset [Sun et al., 2016]. Out of the fourteen classes, we select five classes from the dataset, i.e., affair, education, finance, society and sports. For each class, we sample 25,000 texts as training set and 5,000 as validation set. (Section 4.1) |
| Hardware Specification | No | The paper does not specify any hardware details such as GPU models, CPU types, or memory used for running the experiments. |
| Software Dependencies | No | The paper mentions several software tools like "jieba", "word2vec", "python-pinyin", and a synonym tool. However, it does not provide specific version numbers for these software dependencies, which are necessary for reproducible descriptions. |
| Experiment Setup | No | The paper mentions using CNN and LSTM models and discusses some aspects of the evaluation setup (NLP tasks, datasets, metrics). However, it does not provide specific experimental setup details such as hyperparameter values (e.g., learning rates, batch sizes, number of epochs, optimizers) or other training configurations. |