BESA: BERT-based Simulated Annealing for Adversarial Text Attacks
Authors: Xinghao Yang, Weifeng Liu, Dacheng Tao, Wei Liu
IJCAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on five popular datasets manifest the superiority of BESA compared with existing methods, including Text Fooler, BAE, BERT-Attack, PWWS, and PSO. and 4 Experiments and Analysis |
| Researcher Affiliation | Collaboration | 1School of Computer Science, University of Technology Sydney, Australia 2College of Control Science and Engineering, China University of Petroleum (East China), China 3JD Explore Academy, JD.com, China |
| Pseudocode | Yes | Algorithm 1: The proposed BESA algorithm |
| Open Source Code | Yes | We provide the source code in the supplementary material to ensure the results in this section are reproducible. |
| Open Datasets | Yes | We conduct experiments on five widely used text datasets, including IMDB [Maas et al., 2011] , Question Natural Language Inference (QNLI) [Rajpurkar et al., 2016], Movie Review (MR) [Pang and Lee, 2005], Stanford Natural Language Inference (SNLI) [Bowman et al., 2015], and SST-2 [Socher et al., 2013]. |
| Dataset Splits | Yes | Table 1: Statistic information of the five datasets. # Words denotes the average number of words (i.e., average text length). Dataset # Train # Test # Valid. # Classes # Words IMDB 25,000 25,000 2 235.73 QNLI 104,743 5,463 5,463 2 36.68 MR 8,530 1,066 1,066 2 18.49 SNLI 550,152 10,000 10,000 3 20.27 SST-2 67,349 1,821 872 2 8.67 |
| Hardware Specification | Yes | We carry out all experiments on Enterprise Linux Workstation 7.7 with Intel(R) Xeon(R) Gold 6150 2.7GHz CPU, NVIDIA Quadro P5000 16G GPU, and 176GB RAM. |
| Software Dependencies | No | The paper mentions using specific models (e.g., BERT, Distil BERT, RoBERTa, XLNet) and frameworks (Text Attack), but does not provide specific version numbers for any of these software dependencies. For example, 'Distil BERT' is mentioned, but not 'Distil BERT version X.Y'. |
| Experiment Setup | Yes | The parameter settings for our BESA are given in line 1 of Algorithm 1. Parameter tuning details are listed in supplementary. ... Initialization: USE threshold = 0.5, the highest temperature Tmax = 1000, the lowest temperature Tmin = 50, internal simulation steps K = 20, the balance parameter δ = 0.01, attack radius parameter σ = 3, the initial adversarial example Xadv = X, and the initial time t = 0; |