Neighboring Perturbations of Knowledge Editing on Large Language Models
Authors: Jun-Yu Ma, Zhen-Hua Ling, Ningyu Zhang, Jia-Chen Gu
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments demonstrate the effectiveness of APP coupling with four editing methods on four LLMs. |
| Researcher Affiliation | Academia | 1NERC-SLIP, University of Science and Technology of China 2Zhejiang University 3University of California, Los Angeles. |
| Pseudocode | No | The paper describes the proposed method and its objectives using mathematical equations and textual explanations, but it does not include a formal pseudocode block or algorithm. |
| Open Source Code | Yes | The code and data are available at https://github.com/mjy1111/PEAK. |
| Open Datasets | Yes | To evaluate the additivity of edited models, a benchmark dubbed as Perturbation Evaluation of Appending Knowledge (PEAK) is constructed... The code and data are available at https://github.com/mjy1111/PEAK. |
| Dataset Splits | Yes | To set the hyperparameters, we additionally created a small validation set. |
| Hardware Specification | No | The paper mentions the large language models used (e.g., GPT-2 XL, LLaMA-2) and alludes to 'limited computing resources', but it does not specify any particular hardware like GPU models, CPU types, or memory specifications. |
| Software Dependencies | No | The paper mentions the use of specific LLMs (e.g., GPT-2 XL, GPT-J, LLaMA-2) and frameworks for editing (e.g., Easy Edit), but it does not provide specific version numbers for underlying software libraries like Python, PyTorch, or CUDA. |
| Experiment Setup | Yes | Appendix B.2, 'Hyperparameters for APP', states: 'Table 8 shows the details of hyperparameters set for different LLMs. Besides, the margin M is set to 2.' Table 8 then lists specific alpha, beta, and gamma values for each method and LLM. |