Feedback Detection for Live Predictors
Authors: Stefan Wager, Nick Chamandy, Omkar Muralidharan, Amir Najmi
NeurIPS 2014 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct a pilot study for our proposed methodology using a predictive system currently deployed as a part of a search engine. [...] Finally, in Section 5 we conduct a pilot study based on a predictive model currently deployed as a part of a search engine. |
| Researcher Affiliation | Collaboration | Stefan Wager, Nick Chamandy, Omkar Muralidharan, and Amir Najmi swager@stanford.edu, {chamandy, omuralidharan, amir}@google.com Stanford University and Google, Inc. |
| Pseudocode | No | The paper describes the steps of the method under 'Our Method in Practice' but does not provide a formal pseudocode block or algorithm listing. |
| Open Source Code | No | The paper does not provide any links to open-source code or explicitly state that the code for their methodology is available. |
| Open Datasets | No | The paper mentions using 'historical data collected from log files' and an 'internal' dataset for a 'predictive system currently deployed as a part of a search engine'. No public access information, links, or citations for a publicly available dataset are provided. |
| Dataset Splits | Yes | Our dataset had on the order of 100,000 data points, half of which were used for fitting the model itself and half of which were used for feedback simulation. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware used for running the experiments. |
| Software Dependencies | No | The paper mentions 'standard R libraries' but does not specify any software names with version numbers. |
| Experiment Setup | Yes | We generated data for 5 simulated time periods, adding noise with σ = 0.1 at each step, and fit feedback using a spline basis discussed in Appendix B. The true feedback curve was obtained by fitting a spline regression to the additive feedback model by looking at the unobservable ˆy(t+1) i [?]; we used a df = 5 natural spline with knots evenly spread out on [ 9, 3] in log-odds space plus a jump at 0. [...] The error bars for estimated feedback were obtained using a non-parametric bootstrap [11] for which we resampled pairs of (current, next) predictions. |