FastFill: Efficient Compatible Model Update
Authors: Florian Jaeckle, Fartash Faghri, Ali Farhadi, Oncel Tuzel, Hadi Pouransari
ICLR 2023 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that previous backfilling strategies suffer from decreased performance and demonstrate the importance of both the training objective and the ordering in online partial backfilling. We propose a new training method for feature alignment between old and new embedding models using uncertainty estimation. Compared to previous works, we obtain significantly improved backfilling results on a variety of datasets: m AP on Image Net (+4.4%), Places-365 (+2.7%), and VGG-Face2 (+1.3%). |
| Researcher Affiliation | Collaboration | Florian Jaeckle University of Oxford Fartash Faghri Apple Ali Farhadi Apple Oncel Tuzel Apple Hadi Pouransari Apple |
| Pseudocode | No | The paper describes methods and algorithms textually and through mathematical equations, but it does not include any distinct pseudocode blocks or figures explicitly labeled as algorithms. |
| Open Source Code | Yes | Code available at https://github.com/apple/ml-fct. |
| Open Datasets | Yes | Image Net-1k (21). Image Net-1k is a large-scale image recognition dataset and the most used subset of Image Net [ILSVRC]. It was used in the ILSVRC 2012 image recognition challenge. The dataset contains 1000 object classes and a total of 1,281,167 training images and 50,000 validation images. [...] Places-365 (32). Places-365 is a large-scale scene recognition dataset that spans 365 classes and contains 1.8 million training images and 36,500 validation ones. [...] VGGFace2 (2). VGGFace2 is a large-scale facial recognition dataset. Its training dataset is made up of 8631 classes, each representing a different person, and a total of 3.14 million images. |
| Dataset Splits | Yes | Image Net-1k... contains 1000 object classes and a total of 1,281,167 training images and 50,000 validation images. The training dataset is relatively balanced with about 1.2k images per class. The validation set has the same classes as the training set and exactly 500 images per class. [...] Places-365... contains 1.8 million training images and 36,500 validation ones. The training set contains between 3068 and 5000 images per category and the validation set has exactly 100 images per class. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., CPU, GPU models, or cloud instance types) used for running the experiments. |
| Software Dependencies | No | The paper mentions software components like "Res Net V1.5 Nvidia (17)" which refers to a PyTorch implementation, and optimizers like "SGD" and "Adam Kingma & Ba (13)", but it does not specify explicit version numbers for these software dependencies (e.g., PyTorch 1.x or Python 3.x). |
| Experiment Setup | Yes | We set the embedding dimension of both models to 128. We train both models using the hyper-parameters in Res Net V1.5 Nvidia (17) (SGD optimizer, epochs=100, batchsize=1024, learning rate=1.024, weight decay=3.0517578125 10 5, momentum=0.875, cosine learning rate decay with 5 epochs of linear warmup). For our method, we jointly train a transformation model h, and an uncertainty predictor ψ on Image Net-1k using the objective function defined above (5). We use the same architecture for the MLP architecture for the transformation function h as (20) and the same learning set-up (Adam Kingma & Ba (13), epochs=80, learning rate=5 10 4, cosine learning rate decay with 5 epochs of linear warm-up, freeze of the Batch Norm layers after 40 epochs). |