Convergent Representations of Computer Programs in Human and Artificial Neural Networks
Authors: Shashank Srikant, Ben Lipkin, Anna Ivanova, Evelina Fedorenko, Una-May O'Reilly
NeurIPS 2022 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We investigate this question by analyzing brain recordings derived from functional magnetic resonance imaging (fMRI) studies of programmers comprehending Python code. We first evaluate a selection of static and dynamic code properties, such as abstract syntax tree (AST)-related and runtime-related metrics and study how they relate to neural brain signals. Then, to learn whether brain representations encode fine-grained information about computer programs, we train a probe to align brain recordings with representations learned by a suite of ML models trained on code. |
| Researcher Affiliation | Collaboration | Shashank Srikant 1,4 Benjamin Lipkin 2 Anna A. Ivanova1,2,3 Evelina Fedorenko2,3 Una-May O Reilly1,4 * Equal contribution 1CSAIL, MIT 2 BCS, MIT 3 Mc Govern Institute for Brain Research MIT-IBM Watson AI Lab {shash, lipkinb, annaiv, evelina9}@mit.edu, unamay@csail.mit.edu |
| Pseudocode | No | The paper does not contain any sections or figures explicitly labeled 'Pseudocode' or 'Algorithm', nor does it present structured steps formatted like code. |
| Open Source Code | Yes | We make all the corresponding code, data, and analysis publicly available at https://github.com/ALFA-group/code-representations-ml-brain |
| Open Datasets | Yes | We utilize the publicly available dataset from Ivanova et al. [2020] for all our analyses as it offers high quality, granular brain response data on code comprehension stimuli controlled across multiple code properties. We use the publicly available brain recordings released as part of the study by Ivanova et al. [2020] (MIT license). |
| Dataset Splits | Yes | For each participant, we train a linear regression/classification model with L2-regularization for on unique cross-validated leave-one-run-out folds of the 72 programs they attempted (or 48 programs when sentences are removed). |
| Hardware Specification | Yes | All experiments in this work were run on a single 8-core laptop in under an hour following setup. |
| Software Dependencies | No | The paper mentions various models and general programming languages (Python), but does not provide specific version numbers for software dependencies or libraries used to run the experiments. |
| Experiment Setup | Yes | See Appendix B for a detailed description of model hyper-parameters, cross-validation settings, and significance testing. |