Exemplar Guided Active Learning
Authors: Jason S. Hartford, Kevin Leyton-Brown, Hadas Raviv, Dan Padnos, Shahar Lev, Barak Lenz
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show experimentally that our algorithm only costs logarithmically more than a hypothetical approach that knows all true label frequencies and show experimentally that incorporating automated search can significantly reduce the number of samples needed to reach target accuracy levels. Our experiments are designed to test whether automated search with embeddings could find examples of very rare classes and to assess the effect of different skew ratios on performance. |
| Researcher Affiliation | Industry | AI21 Labs jasonh@cs.ubc.ca AI21 Labs kevinlb@cs.ubc.ca AI21 Labs hadasr@ai21.com AI21 Labs danp@ai21.com AI21 Labs shaharl@ai21.com AI21 Labs barakl@ai21.com |
| Pseudocode | Yes | Algorithm 1: EGAL: Exemplar Guided Active Learning |
| Open Source Code | No | The paper does not provide any statement or link indicating that the source code for the described methodology is publicly available. |
| Open Datasets | No | Beyond these key contributions, we also present a new Reddit word sense disambiguation dataset, which is designed to evaluate active learning methods for highly skewed label distributions. To address this, we collected a new dataset for evaluating active learning methods for word sense disambiguation. |
| Dataset Splits | No | The paper mentions a 'test set' but does not explicitly define a 'validation' split with percentages or counts. |
| Hardware Specification | No | The paper mentions using BERT embeddings and Huggingface's Transformer library but does not specify any hardware details like GPU/CPU models or memory used for experiments. |
| Software Dependencies | No | All experiments used Scikit Learn (Pedregosa et al., 2011) s multi-class logistic regression classifier... We used Huggingface s Transformer library (Wolf et al., 2019). |
| Experiment Setup | No | All experiments used Scikit Learn (Pedregosa et al., 2011) s multi-class logistic regression classifier with default regularization parameters. |