Model Assessment and Selection under Temporal Distribution Shift
Authors: Elise Han, Chengpiao Huang, Kaizheng Wang
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Theoretical analyses and empirical experiments underscore the adaptivity of our proposed methods to the nonstationarity in data. We conduct simulations to verify our theories and test our algorithms on three real datasets. |
| Researcher Affiliation | Academia | 1Department of Computer Science, Columbia University, New York, NY, United States 2Department of Industrial Engineering and Operations Research, Columbia University, New York, NY, United States 3Data Science Institute, Columbia University, New York, NY, United States. |
| Pseudocode | Yes | Algorithm 1 Adaptive Rolling Window for Mean Estimation (Problem 3.2); Algorithm 2 Adaptive Rolling Window for Model Comparison; Algorithm 3 Single-Elimination Tournament for Model Selection; Algorithm 4 Fixed-Window Model Selection Algorithm |
| Open Source Code | Yes | Our code is available at https://github.com/eliselyhan/ARW. |
| Open Datasets | Yes | The first real dataset we use is the ar Xiv dataset1. ...1https://www.kaggle.com/datasets/ Cornell-University/arxiv. Finally, we test our method using a real-world dataset maintained by the Dubai Land Department2. ...2https://www.dubaipulse.gov.ae/data/ dld-transactions/dld_transactions-open |
| Dataset Splits | Yes | At each time period t, we split Bt into a training set Btr t and a validation set Bva t . ... The size of the validation set Bva t is sampled uniformly from {2, 3, 4}. The size of the training set is Btr t = 3Bva t . The data in each period is randomly split into training, validation and test sets. The training set Btr j has 15 samples, the validation set Bva j has 5 samples and the rest of the samples Bte j are used for testing. The data in each period is randomly split into training, validation and test sets with proportions 60%, 20% and 20%, respectively. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running the experiments. It mentions using 'scikit-learn' and 'xgboost' but no hardware specifications. |
| Software Dependencies | No | The paper mentions 'Random Forest Regressor in the Python library scikit-learn' and 'XGBRegressor in the Python library xgboost' but does not specify their version numbers. |
| Experiment Setup | Yes | For random forest regression, we use the function Random Forest Regressor in the Python library scikit-learn. For XGBoost regression, we use the function XGBRegressor in the Python library xgboost. In both cases, we set random_state = 0 and do not change any other default parameters. |