Word Level Robustness Enhancement: Fight Perturbation with Perturbation
Authors: Pei Huang, Yuting Yang, Fuqi Jia, Minghao Liu, Feifei Ma, Jian Zhang10785-10793
AAAI 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct experiments on two important NLP tasks: text classification and natural language inference. Bi LSTM (Conneau et al. 2017) and BERT (Devlin et al. 2019), which represent two popular architectures of deep neural networks, were used to evaluate our robustness enhancement method under two representative attack algorithms. The experimental results show that our robustness enhancement method FPP achieves better performance in accuracy and defense capability compared with existing defense methods. |
| Researcher Affiliation | Academia | 1State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences (ISCAS), Beijing, 100190, China 2 University of Chinese Academy of Sciences, Beijing, 100049, China 3 Laboratory of Parallel Software and Computational Science, ISCAS, Beijing, 100190, China 4 Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China |
| Pseudocode | Yes | The algorithm is presented in Algorithm 1. |
| Open Source Code | Yes | Code is available at https://github.com/YANG-Yuting/fightperturbation-with-perturbation |
| Open Datasets | Yes | MR (Pang and Lee 2005), IMDB (Maas et al. 2011) and SNLI (Bowman et al. 2015) are chosen as data sets. |
| Dataset Splits | No | The paper states: 'Following (Jin et al. 2020), 90% of the MR data is used as the training set and 10% is the test set.' While it specifies train and test splits for MR, it does not explicitly provide details about a validation set split for any of the datasets (MR, IMDB, SNLI). |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU models, CPU types, or memory amounts used for running the experiments. It only describes the model architectures and training optimizers. |
| Software Dependencies | No | The paper mentions using GloVe word embeddings, BERT models, Adam optimizer, and Stanford POS tagger, but it does not specify version numbers for any of these software components, libraries, or tools. |
| Experiment Setup | Yes | For Bi LSTM, we used a 1-layer bidirectional LSTM with 150 hidden units, and 300-dimensional pre-trained Glo Ve word embeddings. We used the 12-layer based version of BERT model with 768 hidden units and 12 heads, with 110M parameters. All models are trained on cross-entropy loss, and we use Adam as our optimizer. For our defense method, we set λ = 0.5, κ = 0.25 and sample size N is 256. |