Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics
Authors: Manuel Brenner, Florian Hess, Georgia Koppe, Daniel Durstewitz
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We first compared MTF on moderately challenging multimodal DS reconstruction tasks3 to a variety of other possible algorithmic strategies that may be considered, all of which lack one or more key ingredients of our methodology. For this, a training and a test set of 100, 000 time steps were created from a Lorenz-63 and a R ossler system with 1% process noise, and a 6d Lewis-Glass network model (Lewis & Glass, 1992; Gilpin, 2022), all in their chaotic regimes (see Methods A.5 for detailed parameter settings and numerical integration). From the simulated trajectories, we then sampled ordinal and count observations using Eqs. 14 and 17 (with randomly drawn parameters), as well as continuous observations with 10% Gaussian noise. Example reconstructions by MTF are in Fig. 2a and Fig. A10. ... As evidenced in Table 1, MTF outperforms all other possible model setups, and in particular the multimodal SVAE by large margins, reinforcing our point about the difficulty of learning an appropriate posterior in the absence of control-theoretic guidance of the training process (cf. Fig. A6). |
| Researcher Affiliation | Academia | Manuel Brenner 1 2 Florian Hess 1 2 Georgia Koppe * 3 4 Daniel Durstewitz * 1 2 3 ... 1Dept. of Theoretical Neuroscience, Central Institute of Mental Health (CIMH), Medical Faculty Mannheim, Heidelberg University, Germany 2Faculty of Physics and Astronomy, Heidelberg University, Germany 3Interdisciplinary Center for Scientific Computing, Heidelberg University, Germany 4Hector Institute for AI in Psychiatry, CIMH. |
| Pseudocode | No | The paper describes the MTF framework and its components using text and mathematical equations, but it does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | All code created is available at https://github.com/Durstewitz Lab/MTF. |
| Open Datasets | Yes | This same dataset was used previously in the study by Kramer et al. (2022) on multimodal data integration for DS reconstruction, and is openly available at https://github.com/Durstewitz Lab/mm PLRNN. ... We used electrophysiological recordings from the rodent hippocampus and spatial location data (Grosmark et al., 2016), publicly available at https://crcns.org/data-sets/hc/hc-11/about-hc-11. |
| Dataset Splits | No | The paper describes train and test set creation (e.g., 'a training and a test set of 100, 000 time steps were created' and 'Multimodal (position and spike count) data were split into a training and a test set of 4600 time steps each'). It mentions "optimal choice for the TF interval" determined via ablation studies, implying a validation process, but it does not explicitly provide percentages or absolute counts for a dedicated validation dataset split, nor does it explicitly mention a cross-validation setup or predefined validation splits with citations as required by the schema. |
| Hardware Specification | No | The paper mentions 'on a single CPU' in the context of a grid search for rotation matrices, but it does not specify any particular hardware details such as GPU models, CPU models, or memory used for the main training of the models. |
| Software Dependencies | No | The paper mentions several software components like 'scipy.fft', 'scipy.stats.spearmanr', 'torch.distributions.NegativeBinomial', 'torch.nn.RNN', 'torch.nn.TransformerEncoder', 'RAdam (Liu et al., 2020)', 'Julia library DynamicalSystems.jl (Datseris, 2018)', and 'dysts Python package (Gilpin, 2022)', but it does not provide specific version numbers for any of these dependencies. |
| Experiment Setup | Yes | To train the dend PLRNN with MTF, RAdam (Liu et al., 2020) was used with a learning rate scheduler that iteratively reduced the learning rate from 10 3 to 10 5 during training. For each epoch, we randomly sampled sequences of length Tseq = 300 from the total training data with a batch size of 16, except for the f MRI data, where we chose Tseq = 72 due to the short overall length of the data (T = 360 per subject). ... Table B1. Hyperparameter settings for MTF, SVAE, BPTT, GVAE-TF, and MS trained on the Lorenz, R ossler and Lewis-Glass model, and for the experimental f MRI data. For explanation of symbols, see Methods A.3. Dataset M B K τ, L, Tseq λMS ν Lorenz 20 15 15 10 1.0 10 R ossler 20 15 15 10 1.0 15 Lewis-Glass 20 15 15 20 1.0 20 f MRI 30 40 30 7 1.0 1 |