Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
MaskMA: Towards Zero-Shot Multi-Agent Decision Making with Mask-Based Collaborative Learning
Authors: Jie Liu, Yinmin Zhang, Chuming Li, Zhiyuan You, Zhanhui Zhou, Chao Yang, Yaodong Yang, Yu Liu, Wanli Ouyang
TMLR 2024 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive experiments in SMAC reveal Mask MA, with a single model trained on 11 training maps, can achieve an impressive 77.8% average zero-shot win rate on 60 unseen test maps by decentralized execution, while also performing effectively on other types of downstream tasks (e.g., varied policies collaboration, ally malfunction, and ad hoc team play). |
| Researcher Affiliation | Collaboration | Jie Liu 1,3, Yinmin Zhang 2,3, Chuming Li2,3, Zhiyuan You1, Zhanhui Zhou3, Chao Yang3, Yaodong Yang 4, Yu Liu5, Wanli Ouyang1,3 1 Multimedia Laboratory, The Chinese University of Hong Kong. 2 The University of Sydney. 3 Shanghai Artificial Intelligence Laboratory. 4 Institute for AI, Peking University. 5 Sense Time Research. |
| Pseudocode | Yes | A.4 Pseudocode The Pytorch-style implementation of the Mask MA is shown below: |
| Open Source Code | No | The paper does not provide a direct link to a code repository or explicitly state that their code for the described methodology is open-source or available in supplementary materials. |
| Open Datasets | Yes | We evaluate Mask MA s performance using the Star Craft Multi-Agent Challenge (SMAC) benchmark. To validate the zero-shot performance, we evaluate the win rate in a challenging setting, using only 11 maps for training and 60 unseen maps for testing. Extensive experiments demonstrate that our model significantly outperforms the previous state-ofthe-art methods in zero-shot scenarios. We also introduce various downstream tasks to further verify the strong robust generalization ability of Mask MA, including varied policies collaboration, ally malfunction, and ad hoc team play. Mask MA is the first approach that achieves strong notable zero-shot capability for multi-agent decision-making (e.g., 78% win rate in SMAC). We hope this work lays the groundwork for further advancements in multi-agent fundamental models, with potential applications across a wide range of domains. |
| Dataset Splits | Yes | To validate the zero-shot performance, we evaluate the win rate in a challenging setting, using only 11 maps for training and 60 unseen maps for testing. |
| Hardware Specification | Yes | We conduct all experiments with a single A100 GPU. |
| Software Dependencies | Yes | We use the following software versions: Cent OS 7.9, Python 3.8.5, Pytorch 2.0.0 (Paszke et al., 2019), Star Craft II 4.10 (Samvelyan et al., 2019), DI-engine 0.2.0 (engine Contributors, 2021) |
| Experiment Setup | Yes | Our experiments of Mask MA and baseline MADT utilize the same hyperparameters which are detailed in the Appendix. Table 8: Hyperparameters of Mask MA and MADT. It should be noted that both models utilize the exact same set of hyperparameters. |