Git Re-Basin: Merging Models modulo Permutation Symmetries
Authors: Samuel Ainsworth, Jonathan Hayase, Siddhartha Srinivasa
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experimentally, we demonstrate the single basin phenomenon across a variety of model architectures and datasets, including the first (to our knowledge) demonstration of zero-barrier linear mode connectivity between independently trained Res Net models on CIFAR-10. |
| Researcher Affiliation | Academia | Paul G. Allen School of Computer Science and Engineering University of Washington |
| Pseudocode | Yes | Algorithm 1: PERMUTATIONCOORDINATEDESCENT |
| Open Source Code | Yes | Our code is open sourced at https://github.com/samuela/git-re-basin. |
| Open Datasets | Yes | We present results for models trained on MNIST (Le Cun et al., 1998), CIFAR-10 (Krizhevsky, 2009), and Image Net (Deng et al., 2009) in Figure 2. |
| Dataset Splits | No | The paper mentions the datasets used (e.g., MNIST, CIFAR-10, ImageNet) but does not explicitly state the training, validation, and test split percentages or sample counts within the paper's text. |
| Hardware Specification | Yes | On currently available hardware (p3.2xlarge AWS instance with an NVIDIA V100 GPU), we observed the following timing results with Algorithm 1 |
| Software Dependencies | No | The paper mentions various algorithms and normalization techniques (e.g., Adam, SGD, Layer Norm, Batch Norm, Re LU) but does not provide specific version numbers for any software dependencies or libraries used. |
| Experiment Setup | Yes | Optimization was done with Adam and a learning rate of 1e 3. ... Optimization was done with SGD with momentum (momentum set to 0.9). A weight decay regularization term of 5e 4 was applied. A single cosine decay schedule with linear warm-up was used. Learning rates were initialized at 1e 6 and linearly increased to 1e 1 over the span of an epoch. After that point a single cosine decay schedule (Loshchilov & Hutter, 2017) was used for the remainder of training. |