Differentially Private Meta-Learning
Authors: Jeffrey Li, Mikhail Khodak, Sebastian Caldas, Ameet Talwalkar
ICLR 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Empirically, we apply our analysis to the problems of federated learning with personalization and few-shot classification, showing that allowing the relaxation to task-global privacy from the more commonly studied notion of local privacy leads to dramatically increased performance in recurrent neural language modeling and image classification. |
| Researcher Affiliation | Collaboration | Jeffrey Li, Mikhail Khodak, Sebastian Caldas Carnegie Mellon University jwl3@cs.cmu.edu Ameet Talwalkar Carnegie Mellon University & Determined AI |
| Pseudocode | Yes | Algorithm 1: Online version of our (ε, δ)-meta-private parameter-transfer algorithm. |
| Open Source Code | No | The paper mentions using 'tools provided by TensorFlow Privacy' and links to its GitHub, but does not provide a link or statement about releasing the source code for their specific methodology. |
| Open Datasets | Yes | Datasets: We train a LSTM-RNN for next word prediction on two federated datasets: (1) The Shakespeare dataset as preprocessed by (Caldas et al., 2018), and (2) a dataset constructed from 3, 000 Wikipedia articles drawn from the Wiki-3029 dataset (Arora et al., 2019), where each article is used as a different task. For few-shot image classification, we use the Omniglot (Lake et al., 2011) and Mini-Image Net (Ravi and Larochelle, 2017) datasets |
| Dataset Splits | Yes | For Shakespeare, we set the number of tokens per task to 800 tokens, leaving 279 tasks for meta-training, 31 for meta-validation, and 35 for meta-testing. For Wikipedia, we set the number of tokens to 1, 600, which corresponds to having 2, 179 tasks for meta-training, 243 for meta-validation, and 606 for meta-testing. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU or CPU models, memory, or specific machine configurations used for running the experiments. |
| Software Dependencies | No | The paper mentions using 'TensorFlow Privacy' but does not specify its version number or any other software dependencies with version numbers. |
| Experiment Setup | Yes | Hyperparameters: For the language-modeling experiments, we tune the hyperparameters on the set of meta-validation tasks. For both datasets and all versions of the meta-learning algorithm, we tune hyperparameters in a two step process. We first tune all the parameters that are not related to refinement: the meta learning rate, the local (within-task) meta-training learning rate, the maximum gradient norm, and the decay constant. Then, we use the configuration with the best accuracy pre-refinement and then tune the refinement parameters: the refine learning rate, refine batch size, and refine epochs. |