AGRO: Adversarial discovery of error-prone Groups for Robust Optimization
Authors: Bhargavi Paranjape, Pradeep Dasigi, Vivek Srikumar, Luke Zettlemoyer, Hannaneh Hajishirzi
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | On the WILDS benchmark, AGRO results in 8% higher model performance on average on known worst-groups, compared to prior group discovery approaches used with G-DRO. |
| Researcher Affiliation | Collaboration | Bhargavi Paranjape1 Pradeep Dasigi2 Vivek Srikumar2,3 Luke Zettlemoyer1,4 Hannaneh Hajishirzi1,2 1University of Washington, 2Allen Institute of Artificial Intelligence, 3University of Utah 4Meta AI |
| Pseudocode | Yes | Algorithm 1 presents the pseudo-code for AGRO. |
| Open Source Code | Yes | Our code and models are public3. 3https://github.com/bhargaviparanjape/robust-transformers |
| Open Datasets | Yes | We evaluate performance on four datasets in language and vision from the popular WILDS benchmark (Koh et al., 2021) to study in-the-wild distribution shifts. |
| Dataset Splits | Yes | We follow the train/dev/test split in (Sagawa et al., 2019), which results in 206,175 training examples. |
| Hardware Specification | No | The paper mentions training models on 'GPU' but does not provide specific details such as GPU models, CPU models, or cloud computing instance specifications. |
| Software Dependencies | No | The paper mentions transformer-based models like DeBERTa-Base, BEIT, RoBERTa-base, and ViT, but does not provide specific software versions for libraries, programming languages, or operating systems used in the experiments. |
| Experiment Setup | Yes | Hyperparameters For θ, we use transformer-based (Vaswani et al., 2017) pretrained models... Appendix A.4 and A.6.3 further detail hyperparameter tuning, particularly for AGRO-specific hyperparameters α and m. |