What does the Knowledge Neuron Thesis Have to do with Knowledge?
Authors: Jingcheng Niu, Andrew Liu, Zining Zhu, Gerald Penn
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We reassess the Knowledge Neuron (KN) Thesis: an interpretation of the mechanism underlying the ability of large language models to recall facts from a training corpus. ... We find that this thesis is, at best, an oversimplification. Not only have we found that we can edit the expression of certain linguistic phenomena using the same model editing methods but, through a more comprehensive evaluation, we have found that the KN thesis does not adequately explain the process of factual expression. ... In this paper, we re-evaluate the KN thesis by expanding the scope of our assessment to include more complex factual patterns and syntactic phenomena. ... We put the KN thesis to the test under the KN-edit framework by asking three questions: (1) can we localise linguistic phenomena using the same KN-edit method; (2) how do the levels of localisation compare to each other; and (3) are these localisations strong enough to support the KN thesis? ... We assembled two new datasets using PARAREL relations to evaluate our two new criteria (see Appendix E for details). |
| Researcher Affiliation | Academia | Jingcheng Niu14 niu@cs.toronto.edu Andrew Liu2 a254liu@uwaterloo.ca Zining Zhu134 zzhu41@stevens.edu Gerald Penn14 gpenn@cs.toronto.edu 1University of Toronto, 2University of Waterloo, 3Stevens Institute of Technology, 4Vector Institute |
| Pseudocode | No | The paper describes methods and uses mathematical equations, but does not include formal pseudocode or algorithm blocks. |
| Open Source Code | Yes | The code, data and results are publicly available at https://github.com/frankniujc/kn thesis. |
| Open Datasets | Yes | We use the linguistic phenomena collected in BLi MP (Warstadt et al., 2020) for our analysis. ... The corpus PARAREL (Elazar et al., 2021) contains facts formulated as a fill-in-the-blank cloze task and it is curated by experts. |
| Dataset Splits | No | The paper mentions datasets used but does not explicitly provide details about training, validation, and test splits (e.g., percentages or sample counts) for its own experiments. It refers to 'new data annotation' for portability and 'assembled two new datasets' for symmetry/synonymy but no split information. |
| Hardware Specification | No | The paper does not provide specific details about the hardware (e.g., GPU models, CPU types) used for running the experiments. |
| Software Dependencies | No | The paper mentions 'huggingface package (Wolf et al., 2020)' and 'bert-base-cased version of BERT, the base version GPT-2 and 7B parameter version of LLa MA-2'. While it mentions specific model versions, it does not provide version numbers for general software dependencies (e.g., PyTorch, TensorFlow, Python version) required for reproducibility. |
| Experiment Setup | No | The paper states, 'We use the same hyperparameters and settings as suggested by Meng et al. (2022)' for causal tracing and 'we follow their instructions and hyperparameters to conduct our evaluation' for ROME on LLaMA-2. However, it does not explicitly list these hyperparameters or system-level training settings within the paper itself for its own experiments. |