Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Natural Gradient VI: Guarantees for Non-Conjugate Models
Authors: Fangyuan Sun, Ilyas Fatkhullin, Niao He
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This work develops a theoretical foundation for stochastic natural gradient variational inference (NGVI) in the non-conjugate setting. Our main contributions are as follows: ... 5 Experiments Numerical Stability of SNGD and Proj-SNGD. In this experiment, we again focus on Poisson regression as discussed in Figure 1. We consider data (x, y) = (0.9, 24) where y is sampled from Poisson(e4x), fixed initialization of variance parameter σ2 0 = 2, and stochastic initialization of mean parameter µ0 Unif([ 3, 0]). We aim to demonstrate numerical stability of Proj-SNGD and SNGD.5 The results of 10 independent runs are shown in Figure 2. Experiment on MNIST Dataset. In this experiment, we compare non-Euclidean algorithms (Proj-SNGD and SNGD) with Euclidean algorithms (Prox-SGD and Proj-SGD) proposed in [Dom20]. Details of implementation and additional results can be found in Section J. We consider logistic regression on a subset of MNIST dataset [Le C98] with labels 6 or 8 (n = 11769, d = 784). We use mini-batches of size 2000 and set the step size γt = γ0/ t, where γ0 is a hyperparameter to be finetuned. We run the algorithm for 1000 epochs (6000 iterations). The results averaged over 5 independent runs are shown in Figure 3. |
| Researcher Affiliation | Academia | Fangyuan Sun1, Ilyas Fatkhullin1,2,3 Niao He1 1Department of Computer Science, ETH Zurich 2ETH AI Center 3Industrial and Systems Engineering, Georgia Institute of Technology EMAIL, EMAIL, EMAIL F.S. is supported by ETH AI Center, I.F. is funded by ETH AI Center Doctoral Fellowship. |
| Pseudocode | Yes | J Experiment Details and Additional Results J.1 Pseudocode of the Algorithms We first present the pseudocode of Proj-SNGD algorithm proposed in Section 4.1. Algorithm 1 Proj-SNGD |
| Open Source Code | No | Justification: The pseudocode for all algorithms used in this paper can be found in the supplementary material. Model parameters (including step size, batch size, etc.) are also provided in Experiments section and the supplementary material. The code will also be released. Justification: The data is publicly available. The code will be released to enable reproduction of the experimental results. |
| Open Datasets | Yes | Experiment on MNIST Dataset. In this experiment, we compare non-Euclidean algorithms (Proj-SNGD and SNGD) with Euclidean algorithms (Prox-SGD and Proj-SGD) proposed in [Dom20]. Details of implementation and additional results can be found in Section J. We consider logistic regression on a subset of MNIST dataset [Le C98] with labels 6 or 8 (n = 11769, d = 784). We use mini-batches of size 2000 and set the step size γt = γ0/ t, where γ0 is a hyperparameter to be finetuned. We run the algorithm for 1000 epochs (6000 iterations). The results averaged over 5 independent runs are shown in Figure 3. Justification: This paper uses the MNIST dataset, which is provided through Py Torch s built-in torchvision library. The original dataset is properly cited in our work. |
| Dataset Splits | No | Experiment on MNIST Dataset. In this experiment, we compare non-Euclidean algorithms (Proj-SNGD and SNGD) with Euclidean algorithms (Prox-SGD and Proj-SGD) proposed in [Dom20]. Details of implementation and additional results can be found in Section J. We consider logistic regression on a subset of MNIST dataset [Le C98] with labels 6 or 8 (n = 11769, d = 784). We use mini-batches of size 2000 and set the step size γt = γ0/ t, where γ0 is a hyperparameter to be finetuned. We run the algorithm for 1000 epochs (6000 iterations). The results averaged over 5 independent runs are shown in Figure 3. Paper text indicates only the dataset and total sample count, but no specific training/test/validation splits or their percentages are provided in the main text. |
| Hardware Specification | Yes | J.4 Implementation Details All experiments, except for CIFAR-10, are conducted on an Apple M3 Pro CPU. On the MNIST dataset, training for 1000 epochs takes approximately 5 minutes, and on the Madelon dataset it takes about 1 minute. The CIFAR-10 experiment is run on an NVIDIA Ge Force RTX 3090 GPU, requiring roughly 8 minutes for 3000 epochs. |
| Software Dependencies | No | Justification: This paper uses the MNIST dataset, which is provided through Py Torch s built-in torchvision library. The original dataset is properly cited in our work. Paper text mentions 'Py Torch' but does not provide a specific version number for any software dependency. |
| Experiment Setup | Yes | Experiment on MNIST Dataset. In this experiment, we compare non-Euclidean algorithms (Proj-SNGD and SNGD) with Euclidean algorithms (Prox-SGD and Proj-SGD) proposed in [Dom20]. Details of implementation and additional results can be found in Section J. We consider logistic regression on a subset of MNIST dataset [Le C98] with labels 6 or 8 (n = 11769, d = 784). We use mini-batches of size 2000 and set the step size γt = γ0/ t, where γ0 is a hyperparameter to be finetuned. We run the algorithm for 1000 epochs (6000 iterations). The results averaged over 5 independent runs are shown in Figure 3. J.4 Implementation Details In the left panel of Figure 3 (MNIST dataset), we set γ0 = 0.05 for Proj-SNGD and SNGD, and γ0 = 0.01 for Prox-SGD and Proj-SGD. In the left panel of Figure 4 (Madelon dataset), we set γ0 = 0.5 for non-Euclidean algorithms, and γ0 = 0.01 for Euclidean algorithms. In the left panel of Figure 5 (CIFAR-10 dataset), we set γ0 = 10 3 for non-Euclidean algorithms, and γ0 = 2 10 4 for Euclidean algorithms. |