Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model
Authors: Tianle Li, Jihai Zhang, Yongming Rao, Yu Cheng
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct comprehensive experiments across multiple post-training strategies to evaluate the compositional capabilities of VLMs. |
| Researcher Affiliation | Collaboration | Tianle Li, Jihai Zhang, Yongming Rao, Yu Cheng The Chinese University of Hong Kong, Tencent Hunyuan Research EMAIL |
| Pseudocode | No | The paper describes methodologies such as Supervised Fine-Tuning and Reinforcement Learning with GRPO using mathematical formulations and descriptive text (Section 3.1, 3.2), but does not present any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | https://github.com/ltl3A87/Com PA |
| Open Datasets | Yes | We introduce Com PABench, a diagnostic benchmark that systematically evaluates compositional generalization in VLMs across modalities, reasoning tasks, and distribution shifts. |
| Dataset Splits | Yes | More specifically, we generate 4K samples for each individual type of data in training and 500 samples for evaluation. For instance, for Cross-Model Composition task, we mix 4K PT-GR and 4K PT-SR data to train a VLM, and test it on 500 PT-GR, 500 PT-SR, etc. |
| Hardware Specification | Yes | All experiments are conducted on 4 NVIDIA H100 GPUs. |
| Software Dependencies | No | Our implementation of GRPO is based on the open-source R1-V1. |
| Experiment Setup | Yes | For training configurations, we apply consistent hyperparameters: a per-device batch size of 1, a learning rate of 1e 6, and a total of 1 training epoch. For RL experiments, we generate 8 completions per prompt in training. And we set the scale before KL divergence constraints to 0, as we observe a dramatic performance degradation with KL divergence in the objective of optimization. |