Debiasing Evaluations That Are Biased by Evaluations
Authors: Jingyan Wang, Ivan Stelmakh, Yuting Wei, Nihar B. Shah10120-10128
AAAI 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We now conduct a semi-synthetic experiment using real grading statistics to evaluate our estimator and our crossvalidation algorithm. ... The results are shown in Fig 1. |
| Researcher Affiliation | Academia | 1 School of Computer Science 2 Department of Statistics & Data Science Carnegie Mellon University {jingyanw, stiv}@cs.cmu.edu, ytwei@cmu.edu, nihars@cs.cmu.edu |
| Pseudocode | Yes | Algorithm 1: Cross-validation. Inputs: observations Y , partial ordering O, and set Λ. |
| Open Source Code | No | The paper mentions an extended version available on arXiv, but it does not provide an explicit statement that the source code for the methodology is open-source or a direct link to a code repository. |
| Open Datasets | Yes | We use the grading data from the course Business Statistics in Spring 2020 from Indiana University Bloomington (2020). This course consists of 10 sessions taught by multiple instructors. ... Indiana University Bloomington. 2020. Grade Distribution Database. https://gradedistribution.registrar.indiana.edu/index.php [Online; accessed 30-Sep-2020]. |
| Dataset Splits | Yes | Our algorithm splits the observations {yij}i [d],j [n] into a training set Ωt [d] [n] and a validation set Ωv [d] [n]. It consists of two steps: a data-splitting step (Lines 1-8) and a validation step (Lines 9-19). |
| Hardware Specification | No | The paper does not provide any specific hardware details such as GPU or CPU models used for running the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies with version numbers. |
| Experiment Setup | Yes | Throughout the experiments, we use Λ = {2i : 9 i 5, i Z} {0, }. ... The bias is generated according to the group ordering induced by the fine grades, with a marginal distribution of N(0, σ2), and the noise is generated i.i.d. from N(0, η2). We set η = 1 σ, and vary the choices of σ. |