Approximate Cross-Validation for Structured Models
Authors: Soumya Ghosh, Will Stephenson, Tin D. Nguyen, Sameer Deshpande, Tamara Broderick
NeurIPS 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate the accuracy and computational benefits of our proposed methods on a diverse set of real-world applications. We support both of these results with practical experiments in Section 5. |
| Researcher Affiliation | Collaboration | Soumya Ghosh MIT-IBM Watson AI Lab IBM Research ghoshso@us.ibm.com William T. Stephenson MIT CSAIL MIT-IBM Watson AI Lab wtstephe@mit.edu Tin D. Nguyen MIT CSAIL MIT-IBM Watson AI Lab tdn@mit.edu Sameer K. Deshpande MIT CSAIL MIT-IBM Watson AI Lab sameerd@alum.mit.edu Tamara Broderick MIT CSAIL MIT-IBM Watson AI Lab tbroderick@csail.mit.edu |
| Pseudocode | Yes | We summarize our method and define ACV, with three arguments, in Algorithm 1; we define IJ(wo) := ACV((1T ), x, o). Algorithm 1: Approximate leave-within-structure-out cross-validation for all folds o O |
| Open Source Code | No | The paper does not contain an explicit statement or link indicating that the source code for the described methodology is publicly available. |
| Open Datasets | Yes | We analyze loop sensor data collected every five minutes over a span of 25 weeks from a section of a freeway near a baseball stadium in Los Angeles. We trained the bilstm-crf model on the Co NLL-2003 shared task benchmark [Sang and De Meulder, 2003] using the English subset of the data. Balocchi and Jensen [2019], Balocchi et al. [2019] have recently studied spatial models of crime in the city of Philadelphia. |
| Dataset Splits | Yes | We trained the bilstm-crf model on the Co NLL-2003 shared task benchmark [Sang and De Meulder, 2003] using the English subset of the data and the pre-defined train/validation/test splits containing 14,987(=N)/3,466/3,684 sentence annotation pairs. |
| Hardware Specification | Yes | Wall-clock time for exact and approximate CV measured on a 2.5GHz quad core Intel i7 processor with 16GB of RAM |
| Software Dependencies | No | The paper mentions using 'stochastic gradient methods' and 'automatic differentiation tools' (citing Baydin et al., 2018) for their implementation. However, it does not specify version numbers for any software libraries, programming languages, or environments (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | No | The paper mentions 'optimize the full model using stochastic gradient methods and employ early stopping by monitoring loss on the validation set' and 'add a small (10 5) regularizer to the diagonal'. However, it explicitly states 'See Appendix L.2 for model architecture and optimization details', indicating that the main text does not contain all specific experimental setup details like concrete hyperparameter values for training. |