Online List Labeling with Predictions
Authors: Samuel McCauley, Ben Moseley, Aidin Niaparast, Shikha Singh
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Finally, the theoretical results are demonstrated empirically. In particular, we show that our data structure performs well on numerous real datasets, including temporal datasets where predictions are constructed from elements that arrived in the past (as is typically done in a practical use case). This section presents experimental results on real data sets. |
| Researcher Affiliation | Academia | Samuel Mc Cauley Department of Computer Science Williams College Williamstown, MA 01267 sam@cs.williams.edu Benjamin Moseley Tepper School of Business Carnegie Mellon University Pittsburgh, PA 15213 moseleyb@andrew.cmu.edu Aidin Niaparast Tepper School of Business Carnegie Mellon University Pittsburgh, PA 15213 aniapara@andrew.cmu.edu Shikha Singh Department of Computer Science Williams College Williamstown, MA 01267 shikha@cs.williams.edu |
| Pseudocode | No | The paper describes the data structure and algorithms in detail within the text, but it does not include formal pseudocode blocks or figures labeled as 'Algorithm' or 'Pseudocode'. |
| Open Source Code | No | The paper does not provide any explicit statements about releasing source code or links to a code repository for the methodology described. |
| Open Datasets | Yes | We use datasets from SNAP Large Network Dataset Collection [37]. [37] Jure Leskovec and Andrej Krevl. SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data, June 2014. |
| Dataset Splits | Yes | To generate the predictions for Learned LLA, we use a contiguous subsequence Ltrain of the input in temporal order as our training data. Our test data Ltest is a contiguous subsequence of the input that comes right after Ltrain, again in temporal order. In all cases, we use the first and second 2^17 = 131072 entries as training and test data, respectively. |
| Hardware Specification | No | The paper does not specify any particular hardware (e.g., CPU, GPU models, or cloud resources) used for running the experiments. |
| Software Dependencies | No | The paper does not list specific software dependencies with version numbers (e.g., Python 3.8, PyTorch 1.9). |
| Experiment Setup | Yes | To generate the predictions for Learned LLA, we use a contiguous subsequence Ltrain of the input in temporal order as our training data. Our test data Ltest is a contiguous subsequence of the input that comes right after Ltrain, again in temporal order. We use two different algorithms for obtaining predictions: predictor1(Ltrain, Ltest) and predictor2(Ltrain, Ltest). In this experiment, we first generate predictions by the algorithm described above, and then we sample t percent of the predictions uniformly at random and make their error as large as possible (the predicted rank is modified to 1 or n, whichever is farthest from the current calculated rank). We repeat the experiment five times, each time resampling the dataset, and report the mean and standard deviation of the results of these experiments. |