Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Bandit and Delayed Feedback in Online Structured Prediction
Authors: Yuki Shibukawa, Taira Tsuchiya, Shinsaku Sakaue, Kenji Yamanishi
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Furthermore, we numerically compare the performance of these algorithms, as well as existing ones. (Abstract) H Numerical experiments This section presents the results of numerical experiments for online multiclass classification and multilabel classification under bandit feedback on MNIST and synthetic data. (Appendix H) |
| Researcher Affiliation | Collaboration | Yuki Shibukawa The University of Tokyo Tokyo, Japan EMAIL Taira Tsuchiya The University of Tokyo and RIKEN AIP Tokyo, Japan EMAIL Shinsaku Sakaue Cyber Agent Tokyo, Japan EMAIL Kenji Yamanishi The University of Tokyo Tokyo, Japan EMAIL |
| Pseudocode | Yes | Algorithm 1 Randomized decoding φΩ Algorithm 2 Randomized decoding with uniform exploration (RDUE) ψΩ Algorithm 3 Black-box Online Learning under Delayed feedback (BOLD) |
| Open Source Code | Yes | The code is provided in the supplementary material. |
| Open Datasets | Yes | We also evaluate the algorithms on the MNIST dataset [31], a widely used benchmark of handwritten digit images. |
| Dataset Splits | No | Data generation We describe the procedure for generating synthetic data... The resulting input vector thus has length n = 40n . These input vectors are generated for T rounds. (Appendix H.1.1) We also evaluate the algorithms on the MNIST dataset [31], a widely used benchmark of handwritten digit images. (Appendix H.1.2) No specific train/test/validation splits or cross-validation methods are described for either the synthetic or MNIST datasets. |
| Hardware Specification | Yes | All experiments were run on a system with 16GB of RAM, Apple M3 CPU, and in Python 3.11.7 on a mac OS Sonoma 14.6.1. |
| Software Dependencies | Yes | All experiments were run on a system with 16GB of RAM, Apple M3 CPU, and in Python 3.11.7 on a mac OS Sonoma 14.6.1. |
| Experiment Setup | Yes | As the algorithm ALG for updating the linear estimator, we employ the OGD in Section 3.2. Following [45], we use the learning rate of ηt = B/ q 2(10 8 + Pt i=1 e Gt 2 F) and no projection is performed in OGD... we fixed B = 10 in all experiments... we repeat experiments 20 times. (Appendix H.1) As the algorithm ALG for updating the linear estimator, we employ OGD as described in Section 3.2 with learning rate ηt = B/ q 2(10 8 + Pt i=1 e Gt 2 F) and orthogonal projection... We fixed B = 50 for all experiments... each experiment was repeated 10 times. (Appendix H.2) |