Brain decoding: toward real-time reconstruction of visual perception
Authors: Yohann Benchetrit, Hubert Banville, Jean-Remi King
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our results are threefold: Firstly, our MEG decoder shows a 7X improvement of image-retrieval over classic linear decoders. Second, late brain responses to images are best decoded with DINOv2, a recent foundational image model. Third, image retrievals and generations both suggest that high-level visual features can be decoded from MEG signals, although the same approach applied to 7T f MRI also recovers better low-level features. Overall, these results, while preliminary, provide an important step towards the decoding in real-time of the visual processes continuously unfolding within the human brain. |
| Researcher Affiliation | Collaboration | Yohann Benchetrit1, , Hubert Banville1, , Jean-R emi King1,2 1FAIR, Meta, 2 Laboratoire des Syst emes Perceptifs, Ecole Normale Sup erieure, PSL University {ybenchetrit,hubertjb,jeanremi}@meta.com |
| Pseudocode | No | The paper describes the architecture and training process in text but does not provide structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide any explicit statement or link regarding the release of open-source code for the described methodology. |
| Open Datasets | Yes | We test our approach on the THINGS-MEG dataset (Hebart et al., 2023). Four participants (2 female, 2 male; mean age of 23.25 years), underwent 12 MEG sessions during which they were presented with a set of 22,448 unique images selected from the THINGS database (Hebart et al., 2019), covering 1,854 categories. |
| Dataset Splits | Yes | We use early stopping on a validation set of 15,800 examples randomly sampled from the original training set, with a patience of 10, and evaluate the performance of the model on a held-out test set (see below). Models are trained on a single Volta GPU with 32 GB of memory. We train each model three times using three different random seeds for the weight initialization of the brain module. |
| Hardware Specification | Yes | Models are trained on a single Volta GPU with 32 GB of memory. |
| Software Dependencies | No | The paper mentions "Adam optimizer (Kingma & Ba, 2014)" and "Scikit-learn (Pedregosa et al., 2011)" but does not provide specific version numbers for all key software components (e.g., deep learning frameworks or other relevant libraries) used for the experiments. |
| Experiment Setup | Yes | Cross-participant models are trained on a set of 63,000 examples using the Adam optimizer (Kingma & Ba, 2014) with default parameters (β1=0.9, β2=0.999), a learning rate of 3 10 4 and a batch size of 128. We use early stopping on a validation set of 15,800 examples randomly sampled from the original training set, with a patience of 10, and evaluate the performance of the model on a held-out test set (see below). |