Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Learning When to Advise Human Decision Makers
Authors: Gali Noti, Yiling Chen
IJCAI 2023 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | The results of a large-scale experiment show that our advising approach manages to provide advice at times of need and to significantly improve human decision making compared to fixed, non-interactive, advising approaches. |
| Researcher Affiliation | Academia | Gali Noti1,2 , Yiling Chen1 1Harvard John A. Paulson School of Engineering and Applied Sciences, Harvard University 2The School of Computer Science and Engineering, The Hebrew University of Jerusalem EMAIL |
| Pseudocode | No | No pseudocode or clearly labeled algorithm block found in the paper. |
| Open Source Code | No | The data are available on the authors websites. |
| Open Datasets | Yes | As the risk-assessment component of the algorithmic assistant, we use the model of [Green and Chen, 2019b] that was trained on 47,141 defendant cases from a dataset collected by the U.S. Department of Justice [DOJ., 2014] (top diagram in Figure 1a)... For learning the advising policy, we train a randomforest model on experimental data of human predictions from [Green and Chen, 2019a]. |
| Dataset Splits | No | For learning the advising policy, we train a randomforest model on experimental data of human predictions from [Green and Chen, 2019a]... The training data consist of 6,250 predictions made by 250 human participants, for 500 defendant cases. |
| Hardware Specification | No | No specific hardware details (GPU/CPU models, memory, or detailed computer specifications) are mentioned in the paper. |
| Software Dependencies | No | No specific software dependencies with version numbers are mentioned in the paper (e.g., Python 3.8, PyTorch 1.9, CPLEX 12.4). |
| Experiment Setup | No | The paper describes the experimental treatments and participant numbers but does not provide specific experimental setup details like hyperparameter values, model initialization, or specific optimizer settings. |