InstructEdit: Instruction-Based Knowledge Editing for Large Language Models
Authors: Ningyu Zhang, Bozhong Tian, Siyuan Cheng, Xiaozhuan Liang, Yi Hu, Kouying Xue, Yanjie Gou, Xi Chen, Huajun Chen
IJCAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct experiments on four datasets and observe that Instruct Edit can equip the Editor with the capability for multi-tasking editing... Our experiments reveal that Instruct Edit can enhance the reliability by 14.86% (compared with MEND) on average when editing GPT2-XL. |
| Researcher Affiliation | Collaboration | Ningyu Zhang1, Bozhong Tian1, Siyuan Cheng2, Xiaozhuan Liang2, Yi Hu2, Kouying Xue2, Yanjie Gou2, Xi Chen2, Huajun Chen1, 1 Zhejiang University 2 Tencent |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The project homepage is at https://zjunlp.github.io/project/ Instruct Edit |
| Open Datasets | Yes | To ensure diversity in multi-task editing, we select a range of datasets: Recent [Zhang et al., 2024] for knowledge insertion, Counter Fact [Zhang et al., 2024] for counterfactual generation, and Conv Sent [Mitchell et al., 2022b] for sentiment editing in knowledge updating. [...] Additionally, we utilize a balanced subset, randomly sampled from the original Conv Sent, for multi-task training. |
| Dataset Splits | No | The paper mentions multi-task editing, hold-out unseen tasks, and training/testing, but does not provide explicit numerical train/validation/test splits (e.g., percentages or specific sample counts) needed to reproduce the data partitioning. |
| Hardware Specification | No | The paper specifies the language models used (GPT2-XL and LLaMA-2-Base) but does not provide specific hardware details like GPU models, CPU models, or memory specifications used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies (e.g., library names with version numbers) needed to replicate the experiment. |
| Experiment Setup | No | The paper discusses experimental settings in general terms, such as the models and baselines used, but does not provide specific experimental setup details like hyperparameter values (e.g., learning rate, batch size, number of epochs) or specific training configurations. |