Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
PROSAC: Provably Safe Certification for Machine Learning Models under Adversarial Attacks
Authors: Chen Feng, Ziquan Liu, Zhuo Zhi, Ilija Bogunovic, Carsten Gerner-Beuerle, Miguel Rodrigues
AAAI 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we conduct extensive experiments with PROSAC to certify the performance of various state-of-the-art vision models in the presence of various adversarial attacks; how the framework recovers existing trends relating to the robustness of different models against different adversarial attacks; and how the framework also suggests new trends relating to state-of-the-art model robustness against attacks. |
| Researcher Affiliation | Academia | 1Department of Electronic and Electrical Engineering, University College London 2School of Electronic Engineering and Computer Science, Queen Mary University of London 3Faculty of Laws, University College London 4AI Centre, Department of Electronic and Electrical Engineering, University College London EMAIL, EMAIL, EMAIL |
| Pseudocode | Yes | Algorithm 1: GP-UCB for hyperparameter optimization |
| Open Source Code | No | No explicit statement or link to the source code for PROSAC or its implementation is provided within the paper. The paper only refers to a third-party tool, advertorch, for implementing attackers. |
| Open Datasets | Yes | Datasets We will consider primarily classification tasks on the Image Net-1k dataset (Deng et al. 2009). |
| Dataset Splits | Yes | We follow the common experimental setting in black-box adversarial attacks, using 1,000 images from Image Net-1k (Andriushchenko et al. 2020; Ilyas et al. 2018) to apply our proposed certification procedure. In particular, we take our calibration set to correspond to this dataset. |
| Hardware Specification | No | The paper does not provide specific hardware details (such as GPU/CPU models, processor types, or memory amounts) used for running the experiments. |
| Software Dependencies | No | The paper mentions using 'advertorch' for attacker implementations but does not specify its version number, nor does it provide version numbers for any other key software dependencies like PyTorch. |
| Experiment Setup | Yes | We set α = 0.10 and ζ = 0.05 in our safety certification procedure, per Definition 1. The hyperparameters of each attacker were carefully selected to explore a wide range of configurations. Specifically, detailed range/values of hyperparameters for each attackers are shown in APPENDIX A. ... We choose βt to be 0.1 with hyper-parameter search from β={0.01,0.1,1.0}. |