BrainBERT: Self-supervised representation learning for intracranial recordings
Authors: Christopher Wang, Vighnesh Subramaniam, Adam Uri Yaari, Gabriel Kreiman, Boris Katz, Ignacio Cases, Andrei Barbu
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 3 EXPERIMENTS We run two sets of experiments, pretraining Brain BERT separately on both STFT and superlet representations. Pretraining examples are obtained by segmenting the neural recordings into 5s intervals. ... Table 1: Brain BERT improves the performance of linear decoders across a wide range of tasks. |
| Researcher Affiliation | Academia | 1MIT CSAIL 2CBMM 3Boston Children s Hospital, Harvard Medical School 1{czw,vsub851,yaari,boris,cases,abarbu}@mit.edu 2gabriel.kreiman@tch.harvard.edu |
| Pseudocode | Yes | Algorithm 1 Time-masking procedure |
| Open Source Code | Yes | Code to train models and reproduce the results was submitted as part of the supplementary materials and can be accessed here: https://github.com/czlwang/Brain BERT. |
| Open Datasets | No | Invasive intracranial field potential recordings were collected during 26 sessions from 10 subjects (5 male, 5 female; aged 4-19, µ 11.9, σ 4.6) with pharmacologically intractable epilepsy. Approximately, 4.37 hours of data were collected from each subject; see appendix J. ... Brain BERT is available as a resource to the community along with the data and scripts to reproduce the results presented here. (While the paper states data is available with the code, it doesn't provide a specific, direct access link or formal citation for the dataset itself, which is a requirement for 'Yes' for datasets.) |
| Dataset Splits | Yes | The data was randomly split into a 80/10/10 training/validation set/test set. |
| Hardware Specification | No | No specific hardware details (e.g., GPU/CPU models, memory amounts, or detailed computer specifications) used for running experiments were mentioned in the paper. |
| Software Dependencies | No | The paper mentions software like LAMB optimizer, AdamW optimizer, GeLU activation, ReLU activation, and Librosa, but does not provide specific version numbers for any of these components or for the main programming language/framework used. |
| Experiment Setup | Yes | All layers (N = 6) in the encoder stack are set with the following parameters: dh = 768, H = 12, and pdropout = 0.1. We pretrain the Brain BERT model with the LAMB optimizer (You et al., 2019) and lr = 1e 4. We use a batch size of nbatch = 256, train for 500k steps, and record the validation performance every 1,000 steps. Then, the weights with the best validation performance are retained. ... The classifier is a fully connected linear layer with din = 768, dout = 1 and a sigmoid activation. The model is trained using a binary cross entropy loss. When fine-tuning, we use the Adam W optimizer, with lr = 1e 3 for the classification head and lr = 1e 4 for the Brain BERT weights. When training with frozen Brain BERT weights, we use the Adam W optimizer with lr = 1e 3. All models, both this classifier and the baseline models, are trained for 1,000 updates. |