PMET: Precise Model Editing in a Transformer
Authors: Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, Jie Yu
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the COUNTERFACT and zs RE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. |
| Researcher Affiliation | Academia | National University of Defense Technology Changsha, Hunan 410073 China xiaopeng.lyy@gmail.com, {shashali, ssz614, yj, majun}@nudt.edu.cn, yangjing2036@126.com |
| Pseudocode | No | The paper describes the PMET method using textual descriptions and mathematical equations, but it does not include any explicitly labeled 'Pseudocode' or 'Algorithm' block. |
| Open Source Code | Yes | Our code is available at https://github.com/xpq-tech/PMET. |
| Open Datasets | Yes | Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the COUNTERFACT and zs RE datasets. ... For the datasets, we performed counterfactual update experiments on two datasets: Zero-Shot Relation Extraction (zs RE) (Levy et al. 2017) and COUNTERFACT (Meng et al. 2022a). |
| Dataset Splits | No | The paper discusses evaluation metrics like efficacy, generalization, and specificity, and mentions 'Implementation details are presented in Appendix D' and 'The evaluation metrics for model editing can be found in Appendix B (Appendix will be found in (Li et al. 2023))'. However, the provided text does not explicitly detail the specific training, validation, and test dataset splits by percentages or counts within the main body. |
| Hardware Specification | No | The paper states, 'Our experiments are conducted on GPT-J (6B) and GPTNeo X (20B),' which are language models. However, it does not specify any hardware details such as GPU models, CPU types, or memory used for conducting these experiments. |
| Software Dependencies | No | The paper does not explicitly mention any software dependencies, libraries, or frameworks with specific version numbers (e.g., Python version, PyTorch version, CUDA version) required to replicate the experiments. |
| Experiment Setup | No | The paper refers to 'Implementation details are presented in Appendix D.' However, the main text itself does not provide specific experimental setup details such as hyperparameter values (e.g., learning rate, batch size) or optimizer settings. |