Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Attacking Perceptual Similarity Metrics
Authors: Abhijay Ghildyal, Feng Liu
TMLR 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In our study, we systematically examine the robustness of these metrics to imperceptible adversarial perturbations. Following the two-alternative forced-choice experimental design with two distorted images and one reference image, we perturb the distorted image closer to the reference via an adversarial attack until the metric flips its judgment. We first show that all metrics in our study are susceptible to perturbations generated via common adversarial attacks such as FGSM, PGD, and the One-pixel attack. Next, we attack the widely adopted LPIPS metric using spatial-transformation-based adversarial perturbations (st Adv) in a white-box setting to craft adversarial examples that can effectively transfer to other similarity metrics in a black-box setting. We also combine the spatial attack st Adv with PGD (ℓ -bounded) attack to increase transferability and use these adversarial examples to benchmark the robustness of both traditional and recently developed metrics. |
| Researcher Affiliation | Academia | Abhijay Ghildyal EMAIL Department of Computer Science Portland State University; Feng Liu EMAIL Department of Computer Science Portland State University |
| Pseudocode | Yes | Algorithm 1: PGD attack on Similarity Metrics; Algorithm 2: st Adv attack on LPIPS; Algorithm 3: FGSM attack on Similarity Metrics (refer Appendix B) |
| Open Source Code | Yes | Code is available at https://tinyurl.com/attackingpsm. |
| Open Datasets | Yes | Our study uses the Berkeley-Adobe perceptual patch similarity (BAPPS) dataset, originally used to train a perceptual similarity metric (Zhang et al., 2018b). We adopt the BAPPS validation dataset (Zhang et al., 2018b) for our experiments. We use the Pie APP test dataset (Prashnani et al., 2018) and the CLIC validation dataset (CLIC, 2022). |
| Dataset Splits | Yes | For the validation set, 5 responses per sample were collected. The final human judgment is the average of the responses. The types of distortions in this dataset are traditional, CNN-based, and distortions by real algorithms such as super resolution, frame interpolation, deblurring, and colorization. Human opinions could be divided, i.e., all responses in a sample may not have voted for the same distorted image. In our study, to ensure that the two distorted images in the sample have enough disparity between them, we only select those samples where humans unanimously voted for one of the distorted images. In total, there are 12,227 such samples. The Pie APP test set consists of 40 reference images with 15 distorted images per reference image. Out of these, we only select those triplet samples where the preference for distorted image A over B was > 85% and vice versa. Hence, we end up with 1381 samples for our experiment. The CLIC dataset contains 5220 triplet samples (reference, distorted image A, and distorted image B), and it acts as a test dataset for us since none of the metrics have been trained on it. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper mentions software components like L-BFGS (Liu & Nocedal, 1989) but does not provide specific version numbers for any libraries, frameworks, or operating systems used in the experiments. |
| Experiment Setup | Yes | For the PGD attack, the maximum ℓ -norm perturbation cannot be more than 0.03 as the step size α is 0.001, and the maximum attack iterations is 30. We chose 30 after visually inspecting for the imperceptibility of perturbations on the generated adversarial samples. For the FGSM attack, we select the maximum ϵ as 0.05. For st Adv attack, L = αLrank + βLflow where α is 50 and β is 0.05. We initialize a flow vector with zeros and optimize it using L-BFGS for max_iterations (250). |