Learning Modality Knowledge Alignment for Cross-Modality Transfer
Authors: Wenxuan Ma, Shuang Li, Lincan Cai, Jingxuan Kang
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | In this work, a series of experiments focusing on the source representation quality during transfer are conducted, revealing the connection between larger modality gap and lesser knowledge reuse which means ineffective transfer. Experiments show that out method enables better reuse of source modality knowledge in cross-modality transfer, which leads to improvements upon existing finetuning methods |
| Researcher Affiliation | Academia | 1Beijing Institute of Technology, Beijing, China 2University of Illinois Urbana-Champaign, USA. |
| Pseudocode | Yes | Algorithm 1 Mo NA: Modality Knowledge Alignment. Algorithm 2 Approximation Algorithm for Modality Knowledge Discrepancy. |
| Open Source Code | No | The paper does not provide any statement about releasing code or a link to a code repository. |
| Open Datasets | Yes | We conduct extensive experiments on two cross-modal benchmarks, NAS-Bench-360 (Tu et al., 2022) and PDEBench (Takamoto et al., 2022). We use Co NLL-2003 and CIFAR-10 as the source modality datasets to compute the outer-loop meta loss. CIFAR-100 (Krizhevsky et al., 2009), Spherical (Cohen et al., 2018), Nina Pro (Atzori et al., 2012), FSD50K (Fonseca et al., 2017b), Darcy-Flow (Li et al., 2020), PSICOV (Adhikari, 2020), Cosmic (Zhang & Bloom, 2020), ECG (Clifford et al., 2017), Satellite (Petitjean et al., 2012), Deep SEA (Zhou & Troyanskaya, 2015). |
| Dataset Splits | No | The paper mentions using established benchmarks and training data sizes (e.g., Table 8 'training data 60K'), but it does not provide explicit training, validation, and test dataset splits (e.g., percentages or counts) or specific cross-validation schemes for all experiments. It states setups follow ORCA, but the detailed splits for reproducibility are not given within the paper itself. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory, or cloud instance types) used for running its experiments. |
| Software Dependencies | No | The paper mentions using PyTorch in the bibliography and specific optimizers like Adam W, but it does not provide version numbers for these software components or any other libraries used. |
| Experiment Setup | Yes | The training configurations for vanilla finetuning (Mo NA s second stage) for each task basically follow the setups in ORCA (Shen et al., 2023) and is summarized in table 9. Table 9 and Table 11 provide specific hyperparameters like 'Batch Size', 'Epoch', 'Grad. Accum.', 'Optimizer', 'Learning Rate', and 'Weight Decay' for various tasks. |