An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions
Authors: Mohammad Jalali, Cheuk Ting Li, Farzan Farnia
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples1. |
| Researcher Affiliation | Academia | Mohammad Jalali Department of Electrical and Computer Engineering Isfahan University of Technology mjalali@ec.iut.ac.ir Cheuk Ting Li Department of Information Engineering The Chinese University of Hong Kong ctli@ie.cuhk.edu.hk Farzan Farnia Department of Computer Science and Engineering The Chinese University of Hong Kong farnia@cse.cuhk.edu.hk |
| Pseudocode | No | The paper presents mathematical theorems and derivations but does not include any pseudocode or algorithm blocks. |
| Open Source Code | Yes | 1The code repository is available at https://github.com/mjalali/renyi-kernel-entropy. |
| Open Datasets | Yes | Specifically, we used the synthetic 8-component and 25-component Gaussian mixture datasets in [26] and the following image datasets: CIFAR-10 [27], Tiny-Image Net [28], MS-COCO [29], AFHQ [30], FFHQ [31] and Image Net [32]. |
| Dataset Splits | Yes | Also, to select the bandwidth parameter σ for the Gaussian kernel in the RKE and RRKE scores, we performed cross-validation and chose the smallest bandwidth σ for which the reported score s standard deviation across 5,000 validation samples is below 0.01. |
| Hardware Specification | No | The paper does not explicitly mention specific hardware details (e.g., GPU/CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific version numbers for any software dependencies or libraries used in the experiments. |
| Experiment Setup | Yes | Also, to select the bandwidth parameter σ for the Gaussian kernel in the RKE and RRKE scores, we performed cross-validation and chose the smallest bandwidth σ for which the reported score s standard deviation across 5,000 validation samples is below 0.01. We provide a more detailed discussion of the bandwidth parameter s selection and the resulting variance in the Appendix. |