Optimizing Black-box Metrics with Adaptive Surrogates
Authors: Qijia Jiang, Olaoluwa Adigun, Harikrishna Narasimhan, Mahdi Milani Fard, Maya Gupta
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimental results on classification and ranking problems verify the proposal performs on par with methods that know the mathematical formulation, and adds notable value when the form of the metric is unknown. |
| Researcher Affiliation | Collaboration | 1Stanford University, USA 2University of Southern California, USA 3Google Research USA. |
| Pseudocode | Yes | Algorithm 1 Surrogate Projected Gradient Descent, Algorithm 2 Finite-difference Gradient Estimate, Algorithm 3 Linear Interpolation Gradient Estimate |
| Open Source Code | Yes | Tensor Flow code has been made available.1 1https://github.com/google-research/google-research/ tree/master/adaptive_surrogates |
| Open Datasets | Yes | The datasets we use are listed in Table 1. ... (1) COMPAS, where the goal is to predict recidivism with gender as the protected attribute (Angwin et al., 2016); (2) Adult, where the goal is to predict if a person s income is more than 50K/year, and we take gender as the protected group (Blake and Merz, 1998); (3) Credit Default, where the task is to predict whether a customer would default on his/her credit card payment, and we take gender as the protected group (Blake and Merz, 1998); (4) Business Entity Resolution, a proprietary dataset...; For this task, we experiment with the KDD Cup 2008 breast cancer detection data set (Rao et al., 2008) popularly used in this literature (Kar et al., 2015; Mackey et al., 2018). |
| Dataset Splits | Yes | In each case, we split the data into train-validation-test sets in the ratio 4/9 : 2/9 : 1/3. |
| Hardware Specification | No | The paper does not provide specific hardware details like GPU/CPU models or processor types used for running experiments. |
| Software Dependencies | No | Tensor Flow code has been made available.1" - only mentions TensorFlow without a specific version number, and no other software with versions. |
| Experiment Setup | Yes | We train linear models in all experiments, and tune hyperparameters such as step sizes and the perturbation parameter σ for gradient estimation using a held-out validation set. We run the projected gradient descent with 250 outer iterations and 1000 perturbations. For the projection step, we run 100 iterations of Adagrad. |