Optimal Transport Model Distributional Robustness
Authors: Van-Anh Nguyen, Trung Le, Anh Bui, Thanh-Toan Do, Dinh Phung
NeurIPS 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | To validate the effectiveness of our framework in the aforementioned settings, we conducted extensive experiments, and the results reveal remarkable improvements compared to the baselines. In this section, we present the results of various experiments to evaluate the effectiveness of our proposed method in achieving distribution robustness. These experiments are conducted in three main settings: a single model, ensemble models, and Bayesian Neural Networks. To ensure the reliability and generalizability of our findings, we employ multiple architectures and evaluate their performance using the CIFAR-10 and CIFAR-100 datasets. For each experiment, we report specific metrics that capture the performance of each model in its respective setting. |
| Researcher Affiliation | Collaboration | Van-Anh Nguyen1 Trung Le1 Anh Tuan Bui1 Thanh-Toan Do1 Dinh Phung 1,2 1Department of Data Science and AI, Monash University, Australia 2Vin AI, Vietnam |
| Pseudocode | No | The paper describes the practical methods using mathematical equations and steps (e.g., equations 9, 10, 11, 12), but it does not include a formally labeled 'Pseudocode' or 'Algorithm' block. |
| Open Source Code | Yes | The implementation is provided in https://github.com/anh-ntv/OT_MDR.git |
| Open Datasets | Yes | We employ multiple architectures and evaluate their performance using the CIFAR-10 and CIFAR-100 datasets. To ensure a fair comparison with SAM, we also set ρ1 = 0.05 for CIFAR-10 experiments and ρ1 = 0.1 for CIFAR-100. |
| Dataset Splits | No | The paper mentions using CIFAR-10 and CIFAR-100 datasets and adopting settings from the original SAM paper. While these datasets have standard splits, the paper does not explicitly state the train/validation/test split percentages or sample counts, nor does it cite a specific source for these splits within the paper itself. |
| Hardware Specification | No | The paper does not specify the hardware (e.g., GPU models, CPU types, memory) used for running the experiments. |
| Software Dependencies | No | The paper does not specify software dependencies with version numbers (e.g., specific Python libraries or frameworks like PyTorch, TensorFlow, along with their versions). |
| Experiment Setup | Yes | For consistency with the original SAM paper, we adopted their setting, using ρ = 0.05 for CIFAR-10 experiments and ρ = 0.1 for CIFAR-100 and report the result in Table 1. In our OT-MDR method, we chose different values of ρ for each half of the mini-batch B, and denoted ρ1 for B1 and ρ2 for B2, where ρ2 = 2ρ1 to be simplified (this simplified setting is used in all experiments). Additionally, similar to SAM [18], when computing the gradient θLB2 k θ2 k(θ) , we set the corresponding Hessian matrix to the identity one. Note that in this setup, we utilize the same hyper-parameters as the one-particle setting, but only train for 100 epochs to save time. All experiments are trained three times with different random seeds. |