Dealing with Synthetic Data Contamination in Online Continual Learning
Authors: Maorong Wang, Nicolas MICHEL, Jiafeng Mao, Toshihiko Yamasaki
NeurIPS 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We experimentally show that contaminated datasets might hinder the training of existing online CL methods. Experiments show that our method can significantly alleviate performance deterioration, especially when the contamination is severe. Comprehensive experiments show that ESRM can successfully mitigate the performance deterioration caused by synthetic contamination, especially when the contamination is severe. |
| Researcher Affiliation | Academia | Maorong Wang1 Nicolas Michel1,2 Jiafeng Mao1 Toshihiko Yamasaki1 1The University of Tokyo 2Univ Gustave Eiffel, CNRS, LIGM |
| Pseudocode | Yes | Algorithm 1 Py Torch-like pseudo-code of ES. |
| Open Source Code | Yes | For reproducibility, the source code of our work is available at https://github.com/maorong-wang/ESRM. |
| Open Datasets | Yes | The synthetic data contamination was simulated across four benchmark datasets used in online CL, including CIFAR-10 [24], CIFAR-100 [24], Tiny Image Net [25], and Image Net-100 [13, 20]. |
| Dataset Splits | Yes | CIFAR-10 [24] has ten classes with 50,000 training images and 10,000 test images. The image is 32 × 32 in size. The dataset is split into five disjoint tasks with two classes per task. CIFAR-100 [24] has 100 classes with 50,000 training samples and 10,000 test samples. Image size is 32 × 32. It is split into 10 disjoint tasks with 10 classes per task. Tiny Image Net [25] has 200 classes, 100,000 training samples, and 10,000 test samples. Image size is 64 × 64. The dataset is split into 100 non-overlapping tasks with two classes per task. Image Net-100 [20] is a subset of Image Net-1k [13] dataset. It consists of 100 classes. The dataset is split into 10 disjoint classes with 10 classes per task. |
| Hardware Specification | Yes | All of the experiments are conducted on NVIDIA A100 GPUs. |
| Software Dependencies | No | The paper mentions software components like PyTorch (implied by 'PyTorch-like pseudo-code'), Hugging Face's Diffusers library, and GLIDE's official implementation. It also lists optimizers (SGD, Adam W) and data augmentation strategies. However, specific version numbers for these software dependencies are not provided in the text. |
| Experiment Setup | Yes | Stream batch size is set to 10 and memory batch size is set to 64. For a fair comparison, we conduct a hyperparameter search on CIFAR-100 (Memory Size = 5K) and apply the same hyperparameter to all settings. The exhaustive list of the hyperparameter search is shown in Table 13. |