Challenges and Considerations in the Evaluation of Bayesian Causal Discovery
Authors: Amir Mohammad Karimi Mamaghan, Panagiotis Tigas, Karl Henrik Johansson, Yarin Gal, Yashas Annadani, Stefan Bauer
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through extensive empirical evaluation, we find that many existing metrics fail to exhibit a strong correlation with the quality of approximation to the true posterior |
| Researcher Affiliation | Academia | 1KTH Royal Institute of Technology 2OATML, University of Oxford 3Digital Futures 4Helmholtz AI 5TU Munich. |
| Pseudocode | No | The paper does not contain any clearly labeled pseudocode or algorithm blocks. |
| Open Source Code | No | The paper references public implementations of existing methods (e.g., 'we utilize the public implementation of BCD-Nets 1') but does not state that the authors are releasing their own code for the work described in this paper. |
| Open Datasets | Yes | We sample graphs from Erd os R enyi (ER) (Erd os et al., 1960) and Scale-Free (SF) (Barab asi & Albert, 1999) random graph family along with a linear Gaussian ANM. |
| Dataset Splits | No | The paper mentions 'different numbers of training samples (N = {5, 10, 100, 1000} were generated for different experiments' and 'Negative Log-Likelihood... of held-out observational samples', but does not explicitly define distinct training, validation, and test splits with proportions or counts. |
| Hardware Specification | No | The paper mentions that 'the computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre,' but does not specify exact GPU/CPU models, processor types, or memory details. |
| Software Dependencies | No | The paper refers to using existing implementations of models (e.g., 'we utilize the public implementation of BCD-Nets'), but it does not provide specific version numbers for software dependencies or libraries used in the experiments. |
| Experiment Setup | Yes | For all experiments, we set the σz, α, γz, and γθ to 0.5, 0.02, 5, and 500, respectively, use 50 particles, and run the model for 20k iterations. |