How Linguistically Fair Are Multilingual Pre-Trained Language Models?
Authors: Monojit Choudhury, Amit Deshpande12710-12718
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We do not give any new results but instead scrutinize a few popular and important works on Multi LMs by explicitly calling out the principles of distributive justice entailed by the choices made by the researchers while resolving the MMSP, but not stated as such. As we shall see, most of the work, whenever possible, follow the Pareto-efficiency principle, i.e., choose the model which does as good or better than all others on all languages that were tested for; otherwise, a utilitarian approach is adopted, where a simple unweighted average performance across languages is used as the model selection criterion. |
| Researcher Affiliation | Industry | Monojit Choudhury, Amit Deshpande Microsoft Research Lab India {monojitc, amitdesh}@microsoft.com |
| Pseudocode | No | No pseudocode or algorithm blocks were found in the paper. |
| Open Source Code | No | The paper does not provide any statement or link indicating that source code for their methodology is openly available. |
| Open Datasets | Yes | She decides to test her models on a standard benchmark, say XNLI (Conneau et al. 2018), which has training data for Natural Language Inferencing task in English and test data for 15 languages including English. She observes that A performs better than B on 10 languages, B performs better than A on 3 languages, and on the remaining two, the models perform equally well. |
| Dataset Splits | No | The paper discusses existing dataset splits from other research works (e.g., XNLI), but does not provide specific training/validation/test dataset splits for its own analysis or experiments. It reinterprets results from benchmarks that have their own predefined splits. |
| Hardware Specification | No | The paper does not describe any specific hardware used for its analysis or reinterpretation of results, as it does not report on new experimental runs. |
| Software Dependencies | No | The paper does not provide specific software dependencies or versions for its analytical work, as it does not report on new experimental runs requiring such details. |
| Experiment Setup | No | The paper does not describe any specific experimental setup details, such as hyperparameters or training settings, as it focuses on reinterpreting previously published results rather than conducting new experiments. |