Elicitation for Aggregation
Authors: Rafael Frongillo, Yiling Chen, Ian Kash
AAAI 2015 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Theoretical | We study the problem of eliciting and aggregating probabilistic information from multiple agents. To formalize this, we consider a principal who wishes to learn the distribution of a random variable. Leveraging techniques from Bayesian statistics, we represent confidence as the number of samples an agent has observed, which is quantified by a hyperparameter from a conjugate family of prior distributions. This then allows us to show that if the principal has access to a few samples, she can achieve her aggregation goal by eliciting predictions from agents using proper scoring rules. In particular, with access to one sample, she can successfully aggregate the agents predictions if and only if every posterior predictive distribution corresponds to a unique value of the hyperparameter, a property which holds for many common distributions of interest. When this uniqueness property does not hold, we construct a novel and intuitive mechanism where a principal with two samples can elicit and optimally aggregate the agents predictions. |
| Researcher Affiliation | Collaboration | Rafael M. Frongillo Harvard University raf@cs.berkeley.edu Yiling Chen Harvard University yiling@seas.harvard.edu Ian A. Kash Microsoft Research iankash@microsoft.com |
| Pseudocode | No | The paper describes conceptual mechanisms and mathematical derivations but does not contain any pseudocode or clearly labeled algorithm blocks. |
| Open Source Code | No | The paper does not contain any explicit statements about releasing source code for the methodology described, nor does it provide any links to a code repository. |
| Open Datasets | No | The paper is theoretical and does not conduct experiments with empirical datasets. The 'samples' referred to in the paper are part of the theoretical model, not an empirical dataset used for training. |
| Dataset Splits | No | The paper is theoretical and does not involve empirical experiments with datasets, thus it does not provide details on training, validation, or test dataset splits. |
| Hardware Specification | No | The paper is theoretical and does not describe computational experiments, therefore no specific hardware specifications are mentioned. |
| Software Dependencies | No | The paper focuses on theoretical concepts and does not describe a software implementation or list any specific software dependencies with version numbers. |
| Experiment Setup | No | The paper is theoretical and does not involve empirical experiments, thus there is no experimental setup, hyperparameters, or training configurations described. |