Revisiting precision recall definition for generative modeling
Authors: Loic Simon, Ryan Webster, Julien Rabin
ICML 2019 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the interest of the proposed formulation over the original approach on controlled multi-modal datasets. |
| Researcher Affiliation | Academia | Normandie Univ, UNICAEN, ENSICAEN, CNRS, GREYC. |
| Pseudocode | Yes | Algorithm 1: Classification-based estimation of the Precision-Recall curve. Inputs: Dataset of target/source sample pairs: D = {(Xi, Yi) P Q i.i.d/i {1, . . . , N}}, Parameterization of the PR curve: Λ = {λ1, . . . , λL} Output: PRDΛ {(αλ, βλ)/λ Λ} Algorithm estimate PRCurve(D, Λ) 1 Dtrain, Dtest =create Train Test(D) 2 f =learn Classifier(Dtrain) 3 PRDΛ =estimate PRD(f, Dtest, Λ) 4 return PRDΛ Procedure create Train Test(D) 1 Dtrain = , Dtest = 2 for i {1, . . . , N} do 2 4 Ztrain i = Ui Xi + (1 Ui)Yi 5 Ztest i = (1 Ui)Xi + Ui Yi 6 Dtrain Dtrain {(Ztrain i , Ui)} 7 Dtest Dtest {(Ztest i , 1 Ui)} end 8 return Dtrain, Dtest Procedure estimate PRD(f, Dtest, Λ) 1 f V als = {f(z)/(z, u) Dtest} 2 err Rates = 3 Nj = |{(z, u) Dtest/u = j}|, for j {0, 1} 4 for t f V als do 5 fpr = 1 N1 {(z, u) Dtest/f(z) < t, u = 1} 6 fnr = 1 N0 {(z, u) Dtest/f(z) t, u = 0} 7 err Rates err Rates {(fpr, fnr)} end 9 for λ Λ do 10 αλ = min({λfpr + fnr/(fpr, fnr) err Rates}) 11 PRDΛ PRDΛ {(αλ, αλ 12 return PRDΛ |
| Open Source Code | No | No explicit statement about the release of source code or a link to a code repository was found. |
| Open Datasets | Yes | In all our experiments, we compute the precision-recall curve between the distribution of features of the Inception Network (Szegedy et al., 2016) (or some other network when specified) instead of using raw images (this choice will be discussed later on). We demonstrate that Algorithm 1 is consistent with the expected notion of precision and recall on controlled datasets such as CIFAR-10 and Imagenet. Figure 2 reproduces an experiment proposed by (Sajjadi et al., 2018). It presents the estimated precision-recall curves on distributions made from CIFAR-10 samples. Figure 3 another controlled experiment with Imagenet samples. |
| Dataset Splits | No | The paper mentions splitting data into training and test sets but does not explicitly mention a separate validation set. For example, 'For training the classifiers, we split each set (real and fake images) into 900 training images and 100 test images.' |
| Hardware Specification | No | No specific hardware details (such as GPU or CPU models, or memory specifications) used for running experiments were mentioned in the paper. |
| Software Dependencies | No | The paper mentions using the ADAM algorithm and specific networks like Inception Network and VGG-Face network, but does not provide specific version numbers for these or any other software dependencies. |
| Experiment Setup | Yes | Each linear classifier is trained independently with the ADAM algorithm. We progressively decrease the learning rate starting from 10^-3 for 50 epochs and use a fixed weight decay of 0.1. |