L-C2ST: Local Diagnostics for Posterior Approximations in Simulation-Based Inference
Authors: Julia Linhart, Alexandre Gramfort, Pedro Rodrigues
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | On standard SBI benchmarks, ℓ-C2ST provides comparable results to C2ST and outperforms alternative local approaches such as coverage tests based on highest predictive density (HPD). We further highlight the importance of local evaluation and the benefit of interpretability of ℓ-C2ST on a challenging application from computational neuroscience. Finally, we report empirical results on two SBI benchmark examples to analyze the performance of ℓ-C2ST and a non-trivial neuroscience use-case that showcases the need of a local validation method. |
| Researcher Affiliation | Academia | Julia Linhart Université Paris-Saclay, Inria, CEA Palaiseau 91120, France julia.linhart@inria.fr Alexandre Gramfort Université Paris-Saclay, Inria, CEA Palaiseau 91120, France alexandre.gramfort@inria.fr Pedro L. C. Rodrigues Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK Grenoble 38000, France pedro.rodrigues@inria.fr |
| Pseudocode | Yes | Algorithm 1: ℓ-C2ST training the classifier on data from the joint distribution |
| Open Source Code | Yes | Code is available at https://github.com/Julia Linhart/lc2st. |
| Open Datasets | Yes | On standard SBI benchmarks, ℓ-C2ST provides comparable results to C2ST and outperforms alternative local approaches such as coverage tests based on highest predictive density (HPD). We illustrate ℓ-C2ST on two examples: Two Moons and SLCP. These models have been widely used in previous works from SBI literature [18, 36] and are part of the SBI benchmark [31]. |
| Dataset Splits | Yes | Let Ns be a fixed simulation budget and {(Θn, Xn)}Ns n=1 = Dtrain Dcal with Dtrain Dcal = . The data from Dtrain are used to train an amortized2 approximation q(θ | x) p(θ | x), e.g. via NPE [18], and those from Dcal to diagnose its local consistency [48]. The classifier accuracy is then empirically estimated over 2Nv samples (Nv samples in each class) from a held-out validation dataset Dv generated in the same way as D |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, processor types, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | All experiments were implemented with Python and the sbi package [44] combined with Py Torch [38] and nflows [12] for neural posterior estimation 4. Classifiers on the C2ST framework use the MLPClassifier from scikit-learn [39] with the same parameters as those used in sbibm [31]. |
| Experiment Setup | Yes | We use implementations from the sbibm package [31] to ensure uniform and consistent experimental setups. We follow the same experimental setting from [3], with a uniform prior over physiologically-relevant parameter values and a simulated dataset from the joint distribution including Ntrain = 50 000 training samples for the posterior estimator and Ncal = 10 000 samples to compute the validation diagnostics. |