A Unifying Normative Framework of Decision Confidence
Authors: Amelia Johnson, Michael Buice, Koosha Khalvati
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Here we present a normative framework for modeling decision confidence that is generalizable to various tasks and experimental setups. We further drive the implications of our model from both theoretical and experimental points of view. Specifically, we show that our model maps to the planning as an inference framework where the objective function is maximizing the gained reward and information entropy of the policy. Moreover, we validate our model on two different psychophysics experiments and show its superiority over other approaches in explaining subjects confidence reports. |
| Researcher Affiliation | Collaboration | Amelia M. Johnson Department of Computer Science University of Washington Seattle, WA 98195 ameliamj@uw.edu; Micheal A. Buice Allen Institute Seattle, WA 98109 michaelbu@alleninstitute.org; Koosha Khalvati Allen Institute Seattle, WA 98109 koosha.khalvati@alleninstitute.org |
| Pseudocode | No | The paper provides mathematical equations and descriptive text for its model but does not include any pseudocode blocks or clearly labeled algorithm sections. |
| Open Source Code | Yes | Our analysis code is also available at https://github.com/ameliamj/decision-confidence-model. |
| Open Datasets | Yes | Both datasets are publicly available and accessible in their corresponding papers [11, 23]. |
| Dataset Splits | Yes | The external observational noise was first fit to each subject s choices in symmetric trials through gradient descent... Therefore, the internal observational noise was fit to the subject s choices in a subset of the trials with prior distribution asymmetries using a grid search and a maximum likelihood estimation with a Bernoulli likelihood function. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments, such as GPU/CPU models or memory specifications. The checklist states: "The model that we ran wasn t incredibly computationally expensive; cumulatively, all of the results could be reproduced in a couple of hours on a normal laptop so it wasn t relevant to mention in the main paper for sake of reproducibility." |
| Software Dependencies | No | The paper describes the mathematical models and fitting procedures but does not list any specific software libraries, frameworks, or their version numbers (e.g., Python, PyTorch, NumPy, SciPy). |
| Experiment Setup | Yes | The external observational noise was first fit to each subject s choices in symmetric trials through gradient descent... Therefore, the internal observational noise was fit to the subject s choices in a subset of the trials with prior distribution asymmetries using a grid search and a maximum likelihood estimation with a Bernoulli likelihood function. ...For each confidence model, the confidence criterion, the threshold at which confidence is binarized into low or high , was fit to the subjects confidence reports in trials with asymmetric prior and fully symmetric trials... This fitting process included a grid search and a maximum likelihood estimation with a Bernoulli likelihood function. |