MultiMoDN—Multimodal, Multi-Task, Interpretable Modular Networks
Authors: Vinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, Mary-Anne Hartley
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform four experiments on several benchmark MM datasets across 10 real-world tasks (predicting medical diagnoses, academic performance, and weather), and show that Multi Mod N s sequential MM fusion does not compromise performance compared with a baseline of parallel fusion. |
| Researcher Affiliation | Academia | Vinitra Swamy* EPFL vinitra.swamy@epfl.ch Malika Satayeva* EPFL malika.satayeva@epfl.ch Jibril Frej EPFL jibril.frej@epfl.ch Thierry Bossy EPFL thierry.bossy@epfl.ch Thijs Vogels EPFL thijs.vogels@epfl.ch Martin Jaggi EPFL martin.jaggi@epfl.ch Tanja Käser* EPFL tanja.kaser@epfl.ch Mary-Anne Hartley* Yale, EPFL mary-anne.hartley@yale.edu |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | We provide an application-agnostic open-source framework for the implementation of Multi Mod N: https://github.com/epfl-iglobalhealth/Multi Mod N. |
| Open Datasets | Yes | MIMIC [30] is a set of deidentified electronic medical records... The dataset is a combination of two MIMIC databases: MIMIC-IV v.1.0 [21] and MIMIC-CXR-JPG v.2.0.0 [22]. After gaining authorized access to Physio Net online repository [30], the embedding dataset can be downloaded via this link: https://physionet.org/content/haim-multimodal/ 1.0.1/. This educational time-series dataset comprises 5, 611 students... It is benchmarked in several recent works [31, 32, 33]. The Weather2k dataset, presented in [36], covers features from 1, 866 weather stations... |
| Dataset Splits | Yes | All results represent a distribution of performance estimates on a model trained 5 times with different random weight initializations for the state vector and weights. Each estimate uses a completely independent test set from an 80-10-10 K-Fold train-test-validation split, stratified on one or more of the prediction targets. |
| Hardware Specification | No | The paper does not provide specific hardware details (such as exact GPU/CPU models, processor types, or memory amounts) used for running its experiments. |
| Software Dependencies | No | A taskand modality-agnostic open-source framework Multi Mod N solution has been implemented in Python using Py Torch as the primary machine learning framework. Experiments were conducted using the same architecture in Py Torch (MIMIC) and Tensor Flow (EDU, Weather). |
| Experiment Setup | Yes | Model architectures were selected among the following hyperparameters: state representation sizes [1, 5, 10, 20, 50, 100], batch sizes [8, 16, 32, 64, 128], hidden features [16, 32, 64, 128], dropout [0, 0.1, 0.2, 0.3], and attention [0, 1]. |