Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Scalable Evaluation and Neural Models for Compositional Generalization
Authors: Giacomo Camposampiero, Pietro Barbiero, Michael Hersche, Roger Wattenhofer, Abbas Rahimi
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | an extensive and modern evaluation on the status of compositional generalization in supervised vision backbones, training more than 5000 models; |
| Researcher Affiliation | Collaboration | Giacomo Camposampiero IBM Research Zurich, ETH Zurich EMAIL Pietro Barbiero IBM Research Zurich EMAIL Michael Hersche IBM Research Zurich EMAIL Roger Wattenhofer ETH Zurich EMAIL Abbas Rahimi IBM Research Zurich EMAIL |
| Pseudocode | Yes | The pseudo-code is included in Algorithm 1 and encompasses different stages. |
| Open Source Code | Yes | Our code is available at github.com/IBM/scalable-compositional-generalization. |
| Open Datasets | Yes | In particular, we experiment with d Sprites [14], I-RAVEN [15], Shapes3D [16], CLEVR [17], Cars3D [18], and MPI3D [19]. More details on the datasets can be found in Appendix B.1. |
| Dataset Splits | Yes | To create the compositional OOD splits, we fix all the degrees of freedom of the proposed orthotopic framework and only study compositional generalization for different values of the c parameter. In particular, we define attributewise thresholds for the values of all task-relevant generative factors in each dataset such that the percentage of excluded attribute-pairs is constant ( 60%). This is tightly mirrored in the size of the training and testing splits, as the data samples are equally distributed among different combinations. |
| Hardware Specification | Yes | All experiments were conducted on compute nodes with an AMD EPYC 7763 64-Core CPU, 2TB RAM, and an NVIDIA A100-SXM4 GPU (80GB), running on Red Hat Enterprise Linux 9.4, CUDA 12.4, and Py Torch 2.3.0+cu121. |
| Software Dependencies | Yes | All experiments were conducted on compute nodes with an AMD EPYC 7763 64-Core CPU, 2TB RAM, and an NVIDIA A100-SXM4 GPU (80GB), running on Red Hat Enterprise Linux 9.4, CUDA 12.4, and Py Torch 2.3.0+cu121. |
| Experiment Setup | Yes | All the models are trained using a standard cross-entropy loss on all attribute labels. The model selection is based on a 10% held-out split of the training data (testing in-distribution generalization). We also adjusted some of the hyperparameters of the network, using a higher weight decay (0.1) and a lower learning rate (0.0001) during training, and use two different random seeds (reporting for each step the best accuracy achieved among the two seeds). |