Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
Authors: Ruihan Wu, Chuan Guo, Awni Hannun, Laurens van der Maaten
NeurIPS 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We perform an error decomposition of systems with multiple machine-learning models, which sheds light on the types of errors that can lead to self-defeating improvements. We also present the results of experiments which show that self-defeating improvements emerge in a realistic stereo-based detection system for cars and pedestrians. |
| Researcher Affiliation | Collaboration | Ruihan Wu Cornell University rw565@cornell.edu Chuan Guo Awni Hannun Laurens van der Maaten Facebook AI Research {chuanguo,awni,lvdmaaten}@fb.com |
| Pseudocode | No | The paper does not contain structured pseudocode or algorithm blocks. |
| Open Source Code | No | The paper does not provide concrete access to source code (specific repository link, explicit code release statement, or code in supplementary materials) for the methodology described in this paper. |
| Open Datasets | Yes | We evaluate our system on the KITTI dataset [16, CC BY-NC-SA 3.0], using the training-validation split of [12]. |
| Dataset Splits | Yes | We evaluate our system on the KITTI dataset [16, CC BY-NC-SA 3.0], using the training-validation split of [12]. |
| Hardware Specification | No | The paper does not provide specific hardware details (exact GPU/CPU models, processor types with speeds, memory amounts, or detailed computer specifications) used for running its experiments. |
| Software Dependencies | No | The paper mentions software like PSMNet, Point-RCNN, and Frustum Point Net but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | We experiment with two training loss functions: (1) the depth mean absolute error and (2) the disparity mean absolute error (MAE). Before computing r(d), we perform winsorization on the prediction u(xleft, xright): we remove all points that are higher than 1 meter in the point cloud (where the camera position is the origin). |