Analytical Probability Distributions and Exact Expectation-Maximization for Deep Generative Networks
Authors: Randall Balestriero, Sebastien PARIS, Richard Baraniuk
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate empirically that EM training of DGNs produces greater likelihood than VAE training. Our new framework will guide the design of new VAE AVI that better approximates the true posterior and open new avenues to apply standard statistical tools for model comparison, anomaly detection, and missing data imputation. We now numerically validate the above EM-steps on a simple problem involving data points on a circle of radius 1 in 2D augmented with a Gaussian noise of standard deviation 0.05. We depict the EM-training of a 2-layer DGN with width of 8 against VAE training. |
| Researcher Affiliation | Academia | Randall Balestriero ECE Department Rice University Sébastien Paris Univ Toulon, Aix Marseille Univ, CNRS, LIS, Toulon, France Richard G. Baraniuk ECE Department Rice University |
| Pseudocode | Yes | From the above result, we can apply the known form of the Gaussian integral on a polytopal region with fewer than S faces and obtain the form of the integral and moments as provided in Appendix B, where detailed pseudo code is also provided. |
| Open Source Code | Yes | Reproducible code for all experiments and figures are available on Github at https://github.com/ Randall Balestriero/EMDGN.git. |
| Open Datasets | Yes | We also experiment with another unidimensional manifold which is a localized subpart of a cosine function in 2D and a more complicated manfiold that is MNIST constrained to the digit 4. We present the manifolds and the EM versus VAE learned manifolds in Fig. 6 and Fig. 7. |
| Dataset Splits | No | The paper mentions training on various datasets (noisy circle, MNIST) but does not specify the explicit training, validation, or test splits (e.g., percentages or sample counts) used for reproducibility. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU/CPU models, memory) used to run the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | Yes | We now numerically validate the above EM-steps on a simple problem involving data points on a circle of radius 1 in 2D augmented with a Gaussian noise of standard deviation 0.05. We depict the EM-training of a 2-layer DGN with width of 8 against VAE training. In all cases the DGNs have the same architecture with same weight initialization; the dataset is also identical between models with the same noise realizations. We also experiment with another unidimensional manifold which is a localized subpart of a cosine function in 2D and a more complicated manfiold that is MNIST constrained to the digit 4. We present the manifolds and the EM versus VAE learned manifolds in Fig. 6 and Fig. 7. Details of training and additional figures for this experiment are provided in Appendix J. |