Exponential Integration for Hamiltonian Monte Carlo
Authors: Wei-Lun Chao, Justin Solomon, Dominik Michels, Fei Sha
ICML 2015 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We consider various ways to derive Gaussian approximations and conduct extensive empirical studies applying the proposed exponential HMC to several benchmarked learning problems. We compare to state-of-the-art methods for improving leapfrog HMC and demonstrate the advantages of our method in generating many effective samples with high acceptance rates in short running times. We validate the effectiveness of exp HMC with extensive empirical studies on various types of distributions, including Bayesian logistic regression and independent component analysis. We also compare exp HMC to alternatives, such as the conventional leapfrog method and the recently proposed Riemann manifold HMC, demonstrating desirable characteristics in scenarios where other methods suffer. |
| Researcher Affiliation | Academia | Wei-Lun Chao1 WEILUNC@USC.EDU Justin Solomon2 JUSTIN.SOLOMON@STANFORD.EDU Dominik L. Michels2 MICHELS@CS.STANFORD.EDU Fei Sha1 FEISHA@USC.EDU 1Department of Computer Science, University of Southern California, Los Angeles, CA 90089 2Department of Computer Science, Stanford University, 353 Serra Mall, Stanford, California 94305 USA |
| Pseudocode | Yes | Figure 1. Leapfrog integrator. Figure 2. Exponential integrator. |
| Open Source Code | Yes | 1Code: https://github.com/pujols/Exponential-HMC |
| Open Datasets | Yes | We consider five datasets from the UCI repository (Bache & Lichman, 2013): Ripley, Heart, Pima Indian, German credit, and Australian credit. |
| Dataset Splits | No | The paper describes burn-in iterations and sample collection for the MCMC process, but does not provide specific train/validation/test dataset splits (e.g., percentages or counts) for the datasets used (UCI, MNIST, MEG). |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., CPU, GPU models, memory, or cloud instances) used for running the experiments. |
| Software Dependencies | No | All the algorithms are implemented in Matlab to ensure a fair comparison1, especially when we evaluate different approaches for their computational cost. |
| Experiment Setup | Yes | We run 200 iterations for burn-in and collect 1000 samples, with (h, L) = (0.6, 8) for all methods, a fairly large step to encourage exploration of sample space. For d exp HMC, we set (N1, N2) = (50, 20). We take L = 100 and manually set h in leapfrog HMC to achieve acceptance rates in [0.6, 0.9] for each combination of dataset and σ, as suggested by Betancourt et al. (2014a). The same (h, L) is used for exp HMC, d exp HMC, and rm Exp HMC, along with (2h, L/2) and (4h, L/4) to test larger steps. We use parameters for RMHMC and e-RMLMC from their corresponding references. We set (N1, N2) = (500, 250) for d exp HMC and apply the same location-specific metric defined by Girolami & Calderhead (2011) along with (N1, N2) = (500, 500) for rm Exp HMC. For exp HMC, (µ, Σ) come from the Laplace approximation. We set σ = 100 for the Gaussian prior (9). We set L = 50 and manually adjust h in leapfrog HMC to achieve acceptance rates in [0.6, 0.9]. The same (h, L) is used for our methods, along with (2h, L/2) and (4h, L/4) to test larger steps. We only examine d exp HMC, given its tractable strategy for Gaussian approximation. The parameters are set to (N1, N2) = (500, 250). We collect 5000 samples after 5000 iterations of burn-in, repeating for 5 trials. In each iteration, we sample L uniformly from {1, ..., L} to ensure high ESS, as in 5.3. |