Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Unsupervised Learning of Evolving Relationships Between Literary Characters
Authors: Snigdha Chaturvedi, Mohit Iyyer, Hal Daume III
AAAI 2017 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | 5 Empirical Evaluation Evaluating these models is difficult for several reasons. Not only is manually designing a taxonomy of relationship types challenging, judging the quality of a learned relationship sequence is also subjective. Therefore, we first use a manually annotated dataset (assuming binary relationship types) to compare the performance of the various models (Sec.5.2). We then evaluate how our model s performance compares with human judgment in characterizing relationships (Sec.5.3). We also evaluate if the learned relationship categories are semantically coherent (Sec.5.4). Lastly, we compare our model with a previously proposed approach (Iyyer et al. 2016) (Sec. 5.5) 1. |
| Researcher Affiliation | Academia | Snigdha Chaturvedi Department of Computer Science University of Illinois Urbana-Champaign EMAIL Mohit Iyyer Department of Computer Science University of Maryland College Park EMAIL Hal Daum e III Department of Computer Science University of Maryland College Park EMAIL |
| Pseudocode | Yes | The paper describes the generative story for the Globally Aware GHMM using numbered steps (e.g., 'For every vector, ft t {1, 2, 3 . . . T}: 1. Toss a choice variable, ct Bernoulli(γ). 2. If ct = 0, choose rt θ(r| F) 3. If ct = 1 & t = 1, then r1 Categorical(π) 4. If ct = 1 & t > 1, then rt Categorical(φrt 1) ρrt 1rt 5. Emit vector ft N(μrt, Σrt)'). |
| Open Source Code | Yes | 1Supplementary material is available on the first author s webpage. |
| Open Datasets | Yes | We use a dataset of 300 English novel-summaries 2, released by Chaturvedi et al. (2016). Chaturvedi, S.; Srivastava, S.; Daum e III, H.; and Dyer, C. 2016. Modeling evolving relationships between characters in literary novels. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA., 2704 2710. |
| Dataset Splits | Yes | We used ϵ = 0.8 (selected using cross-validation). |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for experiments (e.g., GPU/CPU models, memory). |
| Software Dependencies | No | The paper mentions 'Book NLP pipeline' and 'skip-gram model' but does not provide specific version numbers for software dependencies. |
| Experiment Setup | Yes | The vocabulary size of the input sentences was 10K, and that of the feature-sets extracted from them was 4.2K. To obtain word-embeddings (Sec. 3), we used the skipgram model (Mikolov et al. 2013) trained with D = 200 on a collection of novels 3 from Project Gutenberg 4. Globally Aware GHMM uses the average of featurevectors of all sentences in a sequence as its global feature vector (i.e. F = mean( f1, f2 . . . f T )). We used ϵ = 0.8 (selected using cross-validation). Estimating the covariance matrix Σ degraded performance, which might be due to overfitting (Shinozaki and Kawahara 2007). Hence, we only show results for estimating μr, and we use a fixed diagonal matrix as Σ (with each diagonal entry being 0.01), following previous approaches (Lin et al. 2015). |