No-Regret Online Prediction with Strategic Experts
Authors: Omid Sadeghi, Maryam Fazel
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this section, we evaluate the performance of our proposed algorithms for modular utility functions on a publicly available dataset from a Five Thirty Eight forecasting competition4 in which forecasters make predictions about the match outcomes of the 2022 2023 National Football League (NFL). Before each match, Five Thirty Eight provides information on the past performance of the two opposing teams. Forecasters observe this information and make probabilistic predictions about the likelihood of each team winning the match. Considering that there are 284 different matches in the dataset, we set T = 284. Out of the 9982 forecasters who participated in this competition, only 274 made predictions for every single match. We consider two cases: K = 20 and K = 100. To reduce variance, for each case, we sample 5 groups of K forecasters from the 274 and run FTPL and Online Distorted Greedy (ODG) 10 times. We set m = 5. ... In Figure 1, we plot the average regret 1 t E max S [K]:|S|=m Pt τ=1 fτ(S) Pt τ=1 fτ(Sτ) of the two algorithms over time (along with error bands corresponding to 20th and 80th percentiles) along with that of the Five Thirty Eight aggregated predictions for K = 20 and K = 100 settings. |
| Researcher Affiliation | Academia | Omid Sadeghi Maryam Fazel University of Washington Seattle, WA 98195 {omids,mfazel@uw.edu} |
| Pseudocode | Yes | Algorithm 1 Online distorted greedy algorithm |
| Open Source Code | No | The paper provides a link (https://github.com/fivethirtyeight/nfl-elo-game) to the dataset used for experiments (Five Thirty Eight forecasting competition) but does not provide a link or explicit statement about the availability of the authors' own source code for their proposed methodology. |
| Open Datasets | Yes | In this section, we evaluate the performance of our proposed algorithms for modular utility functions on a publicly available dataset from a Five Thirty Eight forecasting competition4 in which forecasters make predictions about the match outcomes of the 2022 2023 National Football League (NFL). ... 4https://github.com/fivethirtyeight/nfl-elo-game |
| Dataset Splits | No | The paper describes the dataset size (T=284 matches) and the number of experts (K=20, K=100) and experimental runs (10 times for 5 groups), but it does not specify explicit training, validation, or test dataset splits. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU models, CPU types, memory) used to run the experiments. |
| Software Dependencies | No | The paper does not provide specific software dependencies or version numbers (e.g., programming languages, libraries, frameworks) used for the implementation or experiments. |
| Experiment Setup | Yes | We set m = 5. Given that FTPL is only approximately incentivecompatible and according to the result of Theorem 3, the reported beliefs could be 2B η 2B distant from the true beliefs, we add a uniformly random value in the range [ 2B η 2B , 2B η 2B ] to the true beliefs to model this fact. We use the standard Laplace distribution as the perturbation for FTPL. Hence, we set B = 1. For both algorithms, the step size η is chosen according to our theoretical results. |