Reliable Fidelity and Diversity Metrics for Generative Models
Authors: Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, Jaejun Yoo
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We analytically and experimentally show that density and coverage provide more interpretable and reliable signals for practitioners than the existing metrics. and 4. Experiments We empirically assess the proposed density and coverage (D&C) metrics and compare against the improved precision and recall (P&R) (Kynk a anniemi et al., 2019). |
| Researcher Affiliation | Collaboration | 1Clova AI Research, Naver Corp. 2Technische Universit at M unchen, Germany 3 Ecole polytechnique f ed erale de Lausanne (EPFL), Switzerland. |
| Pseudocode | No | No pseudocode or clearly labeled algorithm blocks were found in the paper. |
| Open Source Code | Yes | Code: github.com/clovaai/generative-evaluation-prdc . |
| Open Datasets | Yes | We compare the evaluation metrics on MNIST and sound generation tasks using the random embeddings. and We experiment with fake images from Style GAN on Celeb A (Liu et al., 2015) and LSUN-bedroom (Yu et al., 2015). and We use DCGAN generated images (Radford et al., 2016) on MNIST (Le Cun et al., 1998) and Wave GAN generated spectrograms (Donahue et al., 2019) on Speech Commands Zero Through Nine (SC09) dataset. |
| Dataset Splits | No | The paper evaluates existing and proposed metrics on generated and real samples but does not describe train/validation/test splits for model training. |
| Hardware Specification | No | The paper mentions using the Naver Smart Machine Learning (NSML) platform but does not provide specific details on the hardware (GPU, CPU models, etc.) used for experiments. |
| Software Dependencies | No | The paper does not provide specific version numbers for key software components or libraries used in the experiments. |
| Experiment Setup | Yes | In practice, we choose the hyperparameters to achieve E[coverage] > 0.95. For the sake of symmetry, we first set M = N. We then set M = N = 10 000 to ensure a good approximation E[coverage] ≈ 1 − 1/2k, while keeping the computational cost tractable. k = 5 is then sufficient to ensure E[coverage] ≈ 0.969 > 0.95. |