Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Understanding the Generalization Benefit of Model Invariance from a Data Perspective
Authors: Sicheng Zhu, Bang An, Furong Huang
NeurIPS 2021 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In experiments on multiple datasets, we evaluate sample covering numbers for some commonly used transformations and show that the smaller sample covering number for a set of transformations (e.g., the 3D-view transformation) indicates a smaller gap between the test and training error for invariant models, which verifies our propositions. |
| Researcher Affiliation | Academia | Department of Computer Science University of Maryland, College Park EMAIL |
| Pseudocode | Yes | In experiments, we use modified k-medoids [35] clustering method to find the approximation of N(ϵ, S, ρG) (see Algorithm 1). ... Algorithm 1 Sample Covering Number Estimation |
| Open Source Code | Yes | Code is available at https://github.com/bangann/understanding-invariance. |
| Open Datasets | Yes | We report experimental results on CIFAR-10 [29] and Shape Net [10] in this section |
| Dataset Splits | No | The paper describes sampling training images and reports results for different sample sizes per class, but it does not specify a distinct validation set or the exact percentages/counts for train/validation/test splits. |
| Hardware Specification | No | The paper does not provide any specific details about the hardware (e.g., GPU models, CPU types, memory, or cloud instances) used to conduct the experiments. |
| Software Dependencies | No | The paper mentions using ResNet18 and methods like data augmentation and KL divergence regularization but does not list specific software libraries (e.g., PyTorch, TensorFlow) with their version numbers. |
| Experiment Setup | Yes | We use Res Net18 [25] on both datasets... A simple method to learn invariant models is to do data augmentation. The augmented loss function is Laug(x) = L(f(g(x)))... We use this method on CIFAR-10 and Shape Net... We further enforce the invariance using the invariance regularization loss similar to [48, 51]: L = Lcls(f(x)) + λKL(f(x), f(g(x))). ... Table 3 shows specific λ values. |