Towards a Scalable Reference-Free Evaluation of Generative Models
Authors: Azim Ospanov, Jingwei Zhang, Mohammad Jalali, Xuenan Cao, Andrej Bogdanov, Farzan Farnia
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We extensively evaluate FKEA s numerical performance in application to standard image, text, and video datasets. Our empirical results indicate the method s scalability and interpretability applied to large-scale generative models. |
| Researcher Affiliation | Academia | Department of Computer Science & Engineering, The Chinese University of Hong Kong, Hong Kong Department of Cultural and Religious Studies, The Chinese University of Hong Kong, Hong Kong School of Electrical Engineering and Computer Science, University of Ottawa, Canada |
| Pseudocode | Yes | Algorithm 1 FKEA Algorithm for Computing VENDI and RKE reference-free scores |
| Open Source Code | Yes | The codebase is available at https://github.com/aziksh-ospanov/FKEA. |
| Open Datasets | Yes | We extensively evaluate FKEA s numerical performance in application to standard image, text, and video datasets. Our empirical results indicate the method s scalability and interpretability applied to large-scale generative models. |
| Dataset Splits | No | The paper discusses evaluation using sample sizes (e.g., 'sample size n') and mentions using different datasets, but does not provide specific train/validation/test split percentages, sample counts for each split, or detailed methodologies for how the datasets were partitioned for their experiments. |
| Hardware Specification | Yes | Experiments were conducted on RTX3090 GPUs. |
| Software Dependencies | No | The paper mentions software like PyTorch (implied by the research area) but does not provide specific version numbers for any key software components or libraries. |
| Experiment Setup | Yes | In the experiments, we computed the empirical covariance matrix of 2r-dimensional Fourier features with a Gaussian kernel with bandwidth parameter σ tuned for each dataset, and then applied FKEA approximation for the VENDI1, VENDI1.5, and the RKE (same as VENDI2) scores. (...) The Gaussian kernel bandwidth parameter chosen for RKE, VENDI, FKEA-VENDI and FKEA-RKE is σ = 25 and Fourier features dimension 2r = 16k. |