Classification Accuracy Score for Conditional Generative Models
Authors: Suman Ravuri, Oriol Vinyals
NeurIPS 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments are simple: on Image Net, we use three generative models Big GAN-deep at 256 256 and 128 128 resolutions, HAM with masked self-prediction auxiliary decoder at 128 128 resolution, and VQ-VAE-2 at 256 256 resolution to replace the Image Net training set with a model-generated one, train an image classifier, and evaluate performance on the Image Net validation set. |
| Researcher Affiliation | Industry | Suman Ravuri & Oriol Vinyals Deep Mind London, UK N1C 4AG ravuris, vinyals@google.com |
| Pseudocode | No | No pseudocode or algorithm blocks are present in the paper. The methodology is described in prose within Section 3. |
| Open Source Code | Yes | Furthermore, in order to facilitate better diagnoses of generative models, we open-source the proposed metric. ... We open-source our metric on Google Cloud for others to use. The instructions are given in Appendix B. |
| Open Datasets | Yes | Our experiments are simple: on Image Net, we use three generative models... to replace the Image Net training set with a model-generated one, train an image classifier, and evaluate performance on the Image Net validation set. ... Finally, we also compare CAS for different model classes on CIFAR-10. |
| Dataset Splits | Yes | Our experiments are simple: on Image Net... and evaluate performance on the Image Net validation set. To calculate CAS, we replace the Image Net training set with one sampled from the model, and each example from the original training set is replaced with a model sample from the same class. |
| Hardware Specification | Yes | At the time of writing, one can compute the metric in 10 hours for roughly $15, or in 45 minutes for roughly $85 using TPUs. Moreover, depending on affiliation, one may be able to access TPUs for free using the Tensorflow Research Cloud (TFRC) (https://www.tensorflow. org/tfrc/). |
| Software Dependencies | No | The paper mentions 'Tensorflow Research Cloud (TFRC)' but does not specify any software names with version numbers, such as specific library versions or framework versions, used for implementation or experimentation. |
| Experiment Setup | Yes | Big GAN-deep samples are taken from best truncation parameter of 1.5. ... we calculate CAS, IS, and FID for Big GAN-deep models with input noise distributions truncated at different values (known as the truncation trick ). ... Further details about the experiment can be found in Appendix A.1. |