Measuring the reliability of MCMC inference with bidirectional Monte Carlo
Authors: Roger B. Grosse, Siddharth Ancha, Daniel M. Roy
NeurIPS 2016 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 5 Experiments In order to experiment with BREAD, we extended both Web PPL and Stan to run forward and reverse AIS using the sequence of distributions defined in Eq. (2). The MCMC transition kernels were the standard ones provided by both platforms. Our first set of experiments was intended to validate that BREAD can be used to evaluate the accuracy of posterior inference in realistic settings. As an example of how BREAD can be used to guide modeling and algorithmic decisions, we use it to analyze the effectiveness of different representations of a matrix factorization model in both Web PPL and Stan. The reverse AIS estimates are shown in Fig. 1 and Appendix D. (We do not show the forward AIS estimates because these are unaffected by S.) In all five cases, the reverse AIS curves were statistically indistinguishable. |
| Researcher Affiliation | Academia | Roger B. Grosse Department of Computer Science University of Toronto Siddharth Ancha Department of Computer Science University of Toronto Daniel M. Roy Department of Statistics University of Toronto |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper mentions integrating BREAD into Web PPL and Stan and extending them, but does not provide a specific link or explicit statement about making their own implemented code for BREAD open source. |
| Open Datasets | No | The paper states, "we considered five models based on examples from the Stan manual [Sta], and chose a publicly available real-world dataset for each model," but does not provide specific access information (like a link, DOI, or formal citation with author/year for the dataset) to these publicly available datasets. |
| Dataset Splits | No | The paper does not provide specific dataset split information (like exact percentages, sample counts, or explicit train/validation/test subsets) for reproducibility. It discusses simulated and real-world data but not in terms of standard dataset splits. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper mentions Web PPL [GS] and Stan [CGHL+ p] as software used, but does not provide specific version numbers for these or any other software dependencies. |
| Experiment Setup | Yes | In order to experiment with BREAD, we extended both Web PPL and Stan to run forward and reverse AIS using the sequence of distributions defined in Eq. (2). We fixed the values of all hyperparameters to 1, and set N = 50, K = 5 and D = 25. |