Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Understanding Linearity of Cross-Lingual Word Embedding Mappings
Authors: Xutan Peng, Mark Stevenson, Chenghua Lin, Chen Li
TMLR 2022 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | On a novel crosslingual analogy dataset that covers five representative analogy categories for twelve distinct languages, we carry out experiments which provide direct empirical support for our theoretical claim. ... Our hypothesis is verified both theoretically and empirically. We make a justification that the preservation of analogy encoding should be a sufficient and necessary condition for the linearity of CLWE mappings. To provide empirical validation, we first define indicators to qualify the linearity of the ground-truth CLWE mapping (SLMP) and its preservation of analogy encoding (SPAE). Next, we build a novel cross-lingual word analogy corpus... We then benchmark SLMP and SPAE on three representative series of word embeddings. In all setups tested, we observe a significant correlation between SLMP and SPAE, which provides empirical support for our hypothesis. |
| Researcher Affiliation | Collaboration | Xutan Peng EMAIL Mark Stevenson EMAIL Chenghua Lin EMAIL Department of Computer Science, The University of Sheffield Chen Li EMAIL Applied Research Center, Tencent PCG |
| Pseudocode | Yes | Our algorithm is explained using the example in Fig. 2, where the cardinality of X and P is 8 and 4, respectively. Step 1: Link the end points of the vectors within each word pair, hence our target is to adjust these end points so that all connecting lines not only have equal length but also remain parallel. Step 2: For each vector pair (xα, xβ) P, vectorise its connecting line into an offset vector as vα β = xα xβ. Step 3: As the start points of all such offset vectors are aggregated at 0, seek a vector p that minimises the total transportation cost between the end point of p and those of all offset vectors (again, note they share a start point at 0). Step 4: Perform the transportation so that all offset vectors become p , i.e., (xα, xβ) P, T P (xα) T P (xβ) = p . |
| Open Source Code | Yes | We release our data and code at https://github.com/Pzoom522/xANLG. |
| Open Datasets | Yes | Consequently, we develop x ANLG, which we believe to be the first (publicly available) cross-lingual word analogy corpus. ... We use the popular MUSE dictionary (Lample et al., 2018a) which contains a wide range of language pairs. Two existing collections of analogies are utilised: Google Analogy Test Set (GATS) (Mikolov et al., 2013c) ... Bigger Analogy Test Set (BATS) (Gladkova et al., 2016) ... Multilingual Culture-Independent Word Analogy Datasets (MCIWAD) (Ulčar et al., 2020). ... We release our data and code at https://github.com/Pzoom522/xANLG. |
| Dataset Splits | No | The paper describes the construction of a new dataset (xANLG) and mentions generating analogy completion questions from it, but does not specify any training/test/validation splits for machine learning model training or evaluation. It describes sampling for analysis, not for dataset splits for model training/evaluation. |
| Hardware Specification | No | The paper mentions that the SPAE metric "can be obtained within several minutes on a single CPU," but it does not specify the model or type of CPU used for their experiments. Other hardware mentions refer to systems used in related work, not the authors' own experimental setup. |
| Software Dependencies | No | The paper mentions using "the scipy.optimize.fmin implementation of the Nelder-Mead simplex algorithm (Nelder & Mead, 1965)", but it does not specify the version number of the SciPy library or any other software dependencies with their versions. |
| Experiment Setup | Yes | To reach convergence, with the mean offset vector as the initial guess, we set both the absolute errors in parameter and function value between iterations at 1e4. |