Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Vulnerability-Aware Instance Reweighting For Adversarial Training
Authors: Olukorede Fakorede, Ashutosh Kumar Nirala, Modeste Atsague, Jin Tian
TMLR 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through extensive experiments, we show that our proposed method significantly improves over existing reweighting schemes, especially against strong white and black-box attacks. In this section, we verify the effectiveness of the proposed re-weighting function through extensive experiments on various datasets including CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), SVHN(Netzer et al., 2011), and Tiny Image Net(Deng et al., 2009). We employed Res Net-18 (RN-18) (He et al., 2016) and Wide Res Net-34-10 (WRN-34-10) (He et al., 2016) as the backbone models for exploring the effectiveness of the proposed method on CIFAR-10, while CIFAR-100, SVHN, and Tiny Image Net are evaluated on Res Net-18. We summarize our results on CIFAR-10 using RN-18 and WRN-34-10 in Tables 1 and 2, respectively. Moreover, we report results on CIFAR-100, SVHN, and Tiny Imagenet using RN-18 in Tables 3 and 4. Finally, black-box evaluations are made on trained WRN-34-10, and the results are reported in Table 5. Experiments were repeated four times with different random seeds; the mean and standard deviation are subsequently calculated. We conduct ablation studies on the proposed weight assignment function using Res Net-18 on CIFAR-10. |
| Researcher Affiliation | Academia | Olukorede Fakorede* EMAIL Department of Computer Science Iowa State University Ashutosh Nirala EMAIL Department of Computer Science Iowa State University Modeste Atsague EMAIL Department of Computer Science Iowa State University Jin Tian EMAIL Department of Computer Science Iowa State University |
| Pseudocode | Yes | Algorithm 1 VIR-AT Algorithm. Input: a neural network model with the parameters θ, step sizes κ1 and κ2, and a training dataset D of size n. Output: a robust model with parameters θ 1: for epoch = 1 to num_epochs do 2: for batch = 1 to num_batchs do 3: sample a mini-batch {(xi, yi)}M i=1 from D; mini-batch of size M. 4: for i = 1 to M do 5: x i xi + 0.001 N(0, 1), where N(0, I) is the Gaussian distribution with zero mean and 6: identity variance. 7: for k = 1 to K do 8: x i Q Bϵ(xi)(xi + κ1 sign( x i L(fθ(x i), yi)); Q is a projection operator. 9: end for 10: Sv(xi, yi) α e γfθ(xi)y 11: Sd(x i, xi) KL(fθ(xi) fθ(x i)) 12: wi(xi, x i, yi) Sv(xi, yi) Sd(x i, xi) + β; wi(xi, x i, yi) 1 if epoch 76 13: end for 14: θ θ κ2 θ PM i=1 wi(x i, xi, yi) L(fθ(x i), yi) 15: end for 16: end for |
| Open Source Code | No | The paper does not contain any explicit statements about releasing code, nor does it provide any links to a code repository. The provided Open Review link is for review, not for code. |
| Open Datasets | Yes | In this section, we verify the effectiveness of the proposed re-weighting function through extensive experiments on various datasets including CIFAR-10 (Krizhevsky et al., 2009), CIFAR-100 (Krizhevsky et al., 2009), SVHN(Netzer et al., 2011), and Tiny Image Net(Deng et al., 2009). |
| Dataset Splits | No | The paper states that models were trained on datasets like CIFAR-10, CIFAR-100, SVHN, and Tiny Image Net, but it does not explicitly provide the specific training, validation, or test split percentages or counts used for these datasets within the text. While these are common datasets with standard splits, the paper does not explicitly state them. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used to run the experiments, such as GPU models, CPU types, or memory specifications. It only refers to 'models are trained'. |
| Software Dependencies | No | The paper mentions general software concepts like 'mini-batch gradient descent' and 'SGD optimizer with Nesterov momentum' but does not specify any particular software libraries, frameworks, or their version numbers (e.g., 'PyTorch 1.9' or 'TensorFlow 2.x') used for implementation. |
| Experiment Setup | Yes | The models are trained for 115 epochs, using mini-batch gradient descent with momentum 0.9, batch size 128, weight decay 3.5e-3 (RN-18) and 7e-4 (WRN-34-10). The learning rates are set to 0.01 and 0.1 for RN-18 and WRN-34-10 respectively. In both cases, the learning rates are decayed by a factor of 10 at 75th, and then at 90th epoch. In VIR-AT and VIR-TRADES, we introduced the proposed reweighting function on the 76th epoch following (Zhang et al., 2020). The adversarial examples used during training are obtained by perturbing each image using the Projected Gradient Descent (PGD) (Madry et al., 2018) with the following hyperparameters: l norm ϵ = 8/255, step-size κ = 2/255, and K = 10 iterations. The trade-off hyperparameter 1/λ is set to 6.0 for training WRN-34-10 and 4.0 for RN-18 with TRADES. As recommended by the authors, we set the regularization hyperparameter β to 5.0 for training with MART. The values of constants α and β are heuristically determined and set to 7.0 and 0.007 respectively in VIR-AT and 8.0 and 1.6 in VIR-TRADES. Similarly, we set the value of γ to 10.0 and 3.0 in VIR-AT and VIR-TRADES respectively. We set the value of γ to 3.0 for training Tiny Image Net with VIR-AT. Also, the value of 1/λ is set to 5.0 for training VIR-TRADES. |