Achieving High Accuracy with PINNs via Energy Natural Gradient Descent
Authors: Johannes Müller, Marius Zeinhofer
ICML 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate experimentally that energy natural gradient descent yields highly accurate solutions with errors several orders of magnitude smaller than what is obtained when training PINNs with standard optimizers like gradient descent, Adam or BFGS, even when those are allowed significantly more computation time. We test the energy natural gradient approach on four problems: a PINN formulation of a two-dimensional Poisson equation, a PINN formulation of a five-dimensional Poisson equation, a PINN formulation of a one-dimensional heat equation and a deep Ritz formulation of a one-dimensional, nonlinear elliptic equation. |
| Researcher Affiliation | Academia | 1Max Planck Institute for Mathematics in the Sciences Inselstraße 22, 04103 Leipzig, Germany 2Department of Numerical Analysis and Scientific Computing, Simula Research Laboratory, Kristian Augusts Gate 23, 0164, Oslo, Norway. Correspondence to: Marius Zeinhofer <mariusz@simula.no>, Johannes M uller <jmueller@mis.mpg.de>. |
| Pseudocode | Yes | Algorithm 1 Energy Natural Gradient with Line Search Input: initial parameters θ0 Rp, Nmax for k = 1, . . . , Nmax do Compute L(θ) Rp GE(θ)ij D2E( θiuθ, θjuθ) for i, j = 1, . . . , p EL(θ) G+ E(θ) L(θ) η arg minη [0,1] L(θ η EL(θ)) θk = θk 1 η EL(θ) end for |
| Open Source Code | Yes | The code to reproduce the experiments can be found in the repository https://github.com/Marius Zeinhofer/Natural-Gradient-PINNs-ICML23. |
| Open Datasets | No | The paper defines its own collocation points and functions for specific PDE examples (Poisson equation, heat equation, nonlinear elliptic equation). It does not use or provide access to any external, publicly available datasets for training, validation, or testing in the machine learning sense. |
| Dataset Splits | No | The paper uses generated collocation points for solving PDEs and does not explicitly define or use traditional train/validation/test dataset splits. The term 'validation' is not used in the context of data splits for evaluating models during training. |
| Hardware Specification | Yes | All experiments were run on a single NVIDIA RTX 3080 Laptop GPU in double precision. |
| Software Dependencies | No | For our implementation we rely on the library JAX (Bradbury et al., 2018)... The JAX implementation of the least square solve relies on a singular value decomposition. For the implementation of the BFGS optimizer we rely on the implementation jaxopt.BFGS. Specific version numbers for JAX or jaxopt are not provided. |
| Experiment Setup | Yes | We choose the interval [0, 1] for the line search determining the learning rate... In our experiments, we use a grid search over a logarithmically spaced grid on [0, 1] to determine the learning rate η . We initialize the network s weights and biases according to a Gaussian with standard deviation 0.1 and vanishing mean. Then, we test the performance of Adam with an exponentially decreasing learning rate schedul to prevent oscillations, where we start with an initial learning rate of 10 3 that after 1.5 104 steps starts to decrease by a factor of 10 1 every 104 steps until a minimum learning rate of 10 7 is reached or the maximal amount of iterations is completed. |