UniT: A Unified Look at Certified Robust Training against Text Adversarial Perturbation
Authors: Muchao Ye, Ziyi Yin, Tianrong Zhang, Tianyu Du, Jinghui Chen, Ting Wang, Fenglong Ma
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on widely used text classification datasets further demonstrate the effectiveness of the designed unified framework and the proposed DR loss for improving the certified robust accuracy. |
| Researcher Affiliation | Academia | Muchao Ye1 Ziyi Yin1 Tianrong Zhang1 Tianyu Du2 Jinghui Chen1 Ting Wang3 Fenglong Ma1 1The Pennsylvania State University, 2Zhejiang University, 3Stony Brook University {muchao, zmy5171, tbz5156, jzc5917, fenglong}@psu.edu, zjradty@zju.edu.cn, twang@cs.stonybrook.edu |
| Pseudocode | Yes | Algorithm 1: Prediction and certification by Uni T in Type II Setting |
| Open Source Code | Yes | The implementation code is available at https://github.com/machinelearning4health/ Uni T. |
| Open Datasets | Yes | We conduct experiments on four widely used text classification datasets: (1) IMDB [12], (2) SST2 [20], (3) Yelp [19] and (4) AG [29]. https://ai.stanford.edu/~amaas/data/sentiment/ https://www.kaggle.com/datasets/atulanandjha/stanford-sentiment-treebank-v2-sst2 https://github.com/shentianxiao/language-style-transfer/tree/master/ data/yelp https://www.kaggle.com/datasets/amananandrai/ag-news-classification-dataset |
| Dataset Splits | No | The paper describes train and test set sizes for each dataset (e.g., 'IMDB ... 25,000 train and test samples, respectively.'), but no explicit validation set or its size/percentage is mentioned. |
| Hardware Specification | Yes | When we conduct Type I training with Uni T, for every dataset, we fine-tune the pretrained model with 3 epochs, which usually takes 10 minutes on an Nvidia A6000 GPU. When we conduct Type II training with Uni T, the training takes about 48 hours for both datasets on an Nvidia A100 GPU. |
| Software Dependencies | No | The paper mentions using the 'Transformers [24] library' for tokenization, but it does not specify a version number for this library or any other key software dependencies (e.g., Python, PyTorch versions). |
| Experiment Setup | Yes | During training with the DR loss, we set the hyperparameters ν = 0.1 to keep the Gaussian noise relatively small, α = 0.7 to allow the margin to increase while penalizing l2 norm, and ξ = 0.6 to allow appropriate relaxation. In addition, while calculating the final loss, we set β = 1 to make the MR term and the CE loss have equal weight. In the Type II setting, the extra hyperparameters µ and γ have been studied by [31], so we follow them to set µ = 1 and incrementally increase γ to 4 as the training epoch increases. Also, 'fine-tune the pretrained model with 3 epochs' and 'fine-tune the pretrained model with 110 and 200 epochs' are mentioned. |