Revisiting Classifier Two-Sample Tests
Authors: David Lopez-Paz, Maxime Oquab
ICLR 2017 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We evaluate C2ST on a wide variety of synthetic and real data (Section 4), and compare their performance against multiple state-of-the-art alternatives. Furthermore, we provide examples to illustrate how C2ST can interpret the differences between pairs of samples. In Section 5, we propose the use of classifier two-sample tests to evaluate the sample quality of generative models with intractable likelihoods, such as Generative Adversarial Networks (Goodfellow et al., 2014), also known as GANs. |
| Researcher Affiliation | Collaboration | David Lopez-Paz1, Maxime Oquab1,2 1Facebook AI Research, 2WILLOW project team, Inria / ENS / CNRS |
| Pseudocode | No | The paper describes the steps of C2ST verbally but does not include any pseudocode blocks or clearly labeled algorithm sections. |
| Open Source Code | Yes | The implementation of our experiments is available at https://github.com/lopezpaz/classifier_tests. |
| Open Datasets | Yes | We evaluate the use of two-sample tests for model selection in GANs. To this end, we train a number of DCGANs (Radford et al., 2016) on the bedroom class of LSUN (Yu et al., 2015) and the Labeled Faces in the Wild (LFW) dataset (Huang et al., 2007). |
| Dataset Splits | No | The paper specifies splitting the data into 'disjoint training and testing subsets' but does not explicitly mention or quantify a separate validation split. It does refer to 'model selection (such as cross-validation) on Dtr' but not as a defined split. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments (e.g., GPU models, CPU specifications). |
| Software Dependencies | No | The paper mentions 'Torch7 code' and 'scikit-learn implementation' but does not provide specific version numbers for these or other key software components, which is necessary for reproducibility. |
| Experiment Setup | Yes | C2ST-NN has one hidden layer of 20 Re LU neurons, and trains for 100 epochs using the Adam optimizer (Kingma & Ba, 2015). C2ST-KNN uses k = n1/2 tr nearest neighbours for classification. We use a significance level α = 0.05 across all experiments and tests, unless stated otherwise. We train a number of DCGANs (Radford et al., 2016) on the bedroom class of LSUN (Yu et al., 2015) and the Labeled Faces in the Wild (LFW) dataset (Huang et al., 2007). We reused the Torch7 code of Radford et al. (2016) to train a set of DCGANs for {1, 10, 50, 100, 200} epochs, where the generator and discriminator networks are convolutional neural networks (Le Cun et al., 1998) with {1, 2, 4, 8} gf and {1, 2, 4, 8} df filters per layer, respectively. |