Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
HQA-Attack: Toward High Quality Black-Box Hard-Label Adversarial Attack on Text
Authors: Han Liu, Zhi Xu, Xiaotong Zhang, Feng Zhang, Fenglong Ma, Hongyang Chen, Hong Yu, Xianchao Zhang
NeurIPS 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experimental results on five text classification datasets, three natural language inference datasets and two real-world APIs have shown that the proposed HQA-Attack method outperforms other strong baselines significantly. |
| Researcher Affiliation | Academia | Han Liu Dalian University of Technology Dalian, China EMAIL Zhi Xu Dalian University of Technology Dalian, China EMAIL Xiaotong Zhang Dalian University of Technology Dalian, China EMAIL Feng Zhang Peking University Beijing, China EMAIL Fenglong Ma The Pennsylvania State University Pennsylvania, USA EMAIL Hongyang Chen Zhejiang Lab Hangzhou, China EMAIL Hong Yu Dalian University of Technology Dalian, China EMAIL Xianchao Zhang Dalian University of Technology Dalian, China EMAIL |
| Pseudocode | Yes | The detailed algorithm procedure of HQA-Attack is given in Appendix B. |
| Open Source Code | Yes | The source code and demo are publicly available at https://github.com/HQA-Attack/HQAAttack-demo. |
| Open Datasets | Yes | We conduct experiments on five public text classification datasets MR [28], AG s News [42], Yahoo [42], Yelp [42], IMDB [22], and three natural language inference datasets SNLI [3], MNLI [33], m MNLI [33]. |
| Dataset Splits | No | The paper mentions using '1000 test examples' for experiments, but does not explicitly provide details about training or validation splits, percentages, or cross-validation setup for the datasets used. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU models, CPU types, or memory amounts used for running the experiments. It only implies that experiments were run, but without specifying the machines. |
| Software Dependencies | No | The paper does not list specific software dependencies with version numbers, such as programming languages, libraries, or frameworks (e.g., Python version, TensorFlow/PyTorch version). |
| Experiment Setup | Yes | For the hyperparameters, we consistently set r 5 and k 5 for all the datasets. The detailed parameter investigation is provided in Appendix G. In addition, during the optimization procedure, if we re-optimize the same adversarial example three times and no new better adversarial examples are generated, we randomly go back to the last three or four adversarial example. We do not re-optimize an adversarial example more than two times. |