Learning in Deep Factor Graphs with Gaussian Belief Propagation
Authors: Seth Nabarro, Mark Van Der Wilk, Andrew Davison
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments show that these problems can be efficiently solved with belief propagation (BP), whose updates are inherently local, presenting exciting opportunities for distributed and asynchronous training. Our approach can be scaled to deep networks and provides a natural means to do continual learning: use the BP-estimated posterior of the current task as a prior for the next. On a video denoising task we demonstrate the benefit of learnable parameters over a classical factor graph approach and we show encouraging performance of deep factor graphs for continual image classification. |
| Researcher Affiliation | Academia | 1Dyson Robotics Lab, Imperial College London, UK 2Department of Computer Science, University of Oxford, UK. Correspondence to: <sdn09@ic.ac.uk>. |
| Pseudocode | No | The paper describes algorithms using mathematical equations and text, but does not include structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our Tensor Flow (Abadi et al., 2015) implementation is made available2. 2github.com/sethnabarro/gbp_learning/ |
| Open Datasets | Yes | Next, we assess our method in a supervised learning context, evaluating it on MNIST4. 4yann.lecun.com/exdb/mnist/. Creative Commons Attribution Share Alike 3.0 license. ... We apply our method to the task of denoising the bear video from the DAVIS dataset3 (Perazzi et al., 2016)... 3Creative Commons Attributions 4.0 License |
| Dataset Splits | Yes | All model hyperparameters, including factor strengths σ, were tuned on validation sets generated by randomly subsampling 15% of the training set. Further details of the models and hyperparameter selection are included in App. F. We tuned the step size and number of epochs on a randomly sampled validation set comprising 9, 000 training set examples. |
| Hardware Specification | Yes | Denoising the entire 82frame video with the single layer model took 8mins on a NVIDIA RTX 3090 GPU, and the five layer model took 27mins. |
| Software Dependencies | No | The paper mentions 'TensorFlow' but does not specify its version number or versions for other software dependencies. |
| Experiment Setup | Yes | We run for 300 GBP iterations on each frame with damping factor of 0.8 and dropout factor of 0.6 applied to the factor to variable messages. We run GBP in the training graph (with 4 input/output observations) for 600 iterations. The final convolutional factor graph model is summarised in Table D4. Pixel variables are fixed at their observed values. To produce the results shown in Fig. 4, we train with continual learning, with a batchsize of 50 and run 500 GBP iterations on each batch. At test time, we fix the parameters and run GBP for 300 iterations per test batch of 200 examples. We apply a damping factor of 0.9 and a dropout factor of 0.5 to the factor to variable messages at both train and test time. |