Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
The Unreasonable Effectiveness of Linear Prediction as a Perceptual Metric
Authors: Daniel Severo, Lucas Theis, Johannes BallΓ©
ICLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on full-reference image quality assessment datasets show LASI performs competitively with learned deep feature based methods like LPIPS (Zhang et al., 2018) and PIM (Bhardwaj et al., 2020), at a similar computational cost to hand-crafted methods such as MS-SSIM (Wang et al., 2003). |
| Researcher Affiliation | Collaboration | Daniel Severo University of Toronto Vector Institute for A.I. EMAIL Lucas Theis Google Deepmind EMAIL Johannes Ball e Google Research EMAIL |
| Pseudocode | Yes | 4.2 ALGORITHM Here we describe our implementation which solves (1) in 3 steps. The algorithm is differentiable and most operations can be run in parallel on a GPU. Compute time and memory can be traded-off by, for example, precomputing the rank-one matrices. It is also possible to solve Equation (1) thrice in parallel, once for each channel, and average the results at the expense of some performance on downstream perceptual tasks. The steps of our method are: 1) Transform... 2) Weigh-and-Sum... 3) Solve... |
| Open Source Code | Yes | Code: https://github. com/dsevero/Linear-Autoregressive-Similarity-Index. |
| Open Datasets | Yes | To move away from handcrafting similarity metrics the community has shifted towards using deep features from large pre-trained neural networks. For example, the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) metric assumes the L2 distance between these deep features can capture human perception. In the same work, the authors introduce the Berkeley Adobe Perceptual Patch Similarity (BAPPS) dataset, which has become widely accepted as a benchmark for measuring perceptual alignment of similarity metrics. |
| Dataset Splits | Yes | These methods are trained and evaluated by performing train test splits on benchmark 2-AFC datasets such as the Berkeley Adobe Perceptual Patch Similarity (BAPPS) (Zhang et al., 2018). |
| Hardware Specification | Yes | Figure 3: Left: Total time to compute distances for 1000 examples from BAPPS on an NVIDIA V100. Results are averaged over 100 runs. |
| Software Dependencies | No | The paper mentions 'jax.numpy.linalg.pinv; Bradbury et al., 2018' and implies the use of Python, but does not provide specific version numbers for these or other software dependencies, which are necessary for full reproducibility. |
| Experiment Setup | Yes | For our method we used N = 12 across all experiments and the decay parameter in Equation (1) was fixed to Ο = 0.8, the same value used for the lossless compression algorithm of Meyer & Tischer (2001). Images in all categories have dimensions 64 Γ 64 Γ 3 which is 1536 times larger than the neighborhood size. |