SIG: Speaker Identification in Literature via Prompt-Based Generation
Authors: Zhenlin Su, Liyan Xu, Jin Xu, Jiangnan Li, Mingdu Huangfu
AAAI 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform both cross-domain evaluation and in-domain evaluation on PDNC, the largest dataset of this task, where empirical results suggest that SIG outperforms previous baselines of complicated designs, as well as the zero-shot Chat GPT, especially excelling at those hard non-explicit scenarios by up to 17% improvement. Additional experiments on another dataset WP further corroborate the efficacy of SIG. |
| Researcher Affiliation | Collaboration | 1School of Future Technology, South China University of Technology 2We Chat AI, Tencent 3Pazhou Lab, Guangzhou 4Institute of Information Engineering, Chinese Academy of Sciences |
| Pseudocode | No | The paper describes the approach using text and a figure (Figure 1) but does not include a formally labeled 'Pseudocode' or 'Algorithm' block with structured steps. |
| Open Source Code | No | The paper does not provide any concrete access to source code for the methodology described, such as a specific repository link or an explicit code release statement. |
| Open Datasets | Yes | Our main experiments are conducted on Project Dialogism Novel Corpus (PDNC) (Vishnubhotla, Hammond, and Hirst 2022), a recently introduced dataset specifically designed for analysis on English literary text. ... In addition, SIG is also evaluated on WP (Chen, Ling, and Dai 2019), a speaker identification dataset stemmed from a Chinese novel, and shown surpassing previous baselines by 5.2% on the test set. |
| Dataset Splits | Yes | As PDNC consists of multiple novels, it enables crossdomain evaluation, such that the model is evaluated on the test set that has no overlapped novels from the training. ... For each experiment, four novels are randomly selected as the test set, while the remaining novels are used for training the model. ... Specifically, the training set comprises only explicit quotations, while the remaining quotations (implicit and anaphoric) are assigned to the test set. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory amounts, or detailed computer specifications) used for running its experiments. It only vaguely mentions 'computational resource constraints'. |
| Software Dependencies | Yes | We employ Chat GPT (gpt-3.5-turbo-0613) from Open AI for the zero-shot experiments. ... SIG employs BART (Lewis et al. 2020) as the sequence generation PLM. ... We also adopt Ro BERTa (Liu et al. 2019) as the encoder-only model... |
| Experiment Setup | No | The paper describes prompt template design and the training objective, but it does not provide specific hyperparameter values such as learning rate, batch size, number of epochs, or optimizer settings for reproducibility. |