Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions
Authors: Mohammad Jalali, Cheuk Ting Li, Farzan Farnia
NeurIPS 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Utilizing the RKE score, we conduct an extensive evaluation of state-of-the-art generative models over standard image datasets. The numerical results indicate that while the recent algorithms for training generative models manage to improve the mode-based diversity over the earlier architectures, they remain incapable of capturing the full diversity of real data. Our empirical results provide a ranking of widely-used generative models based on the RKE score of their generated samples1. |
| Researcher Affiliation | Academia | Mohammad Jalali Department of Electrical and Computer Engineering Isfahan University of Technology EMAIL Cheuk Ting Li Department of Information Engineering The Chinese University of Hong Kong EMAIL Farzan Farnia Department of Computer Science and Engineering The Chinese University of Hong Kong EMAIL |
| Pseudocode | No | The paper presents mathematical theorems and derivations but does not include any pseudocode or algorithm blocks. |
| Open Source Code | Yes | 1The code repository is available at https://github.com/mjalali/renyi-kernel-entropy. |
| Open Datasets | Yes | Specifically, we used the synthetic 8-component and 25-component Gaussian mixture datasets in [26] and the following image datasets: CIFAR-10 [27], Tiny-Image Net [28], MS-COCO [29], AFHQ [30], FFHQ [31] and Image Net [32]. |
| Dataset Splits | Yes | Also, to select the bandwidth parameter σ for the Gaussian kernel in the RKE and RRKE scores, we performed cross-validation and chose the smallest bandwidth σ for which the reported score s standard deviation across 5,000 validation samples is below 0.01. |
| Hardware Specification | No | The paper does not explicitly mention specific hardware details (e.g., GPU/CPU models, memory) used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific version numbers for any software dependencies or libraries used in the experiments. |
| Experiment Setup | Yes | Also, to select the bandwidth parameter σ for the Gaussian kernel in the RKE and RRKE scores, we performed cross-validation and chose the smallest bandwidth σ for which the reported score s standard deviation across 5,000 validation samples is below 0.01. We provide a more detailed discussion of the bandwidth parameter s selection and the resulting variance in the Appendix. |