Entropy Minimization In Emergent Languages
Authors: Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, Marco Baroni
ICML 2020 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We establish our results in the context of signaling games (Lewis, 1969)... We find experimentally that, once the agents are successfully trained to jointly solve the task, the emergent protocol minimizes the entropy of the messages or, equivalently in our setup, the mutual information between Sender’s input and messages. |
| Researcher Affiliation | Collaboration | 1Facebook AI Research, Paris, France 2Cognitive Machine Learning (ENS EHESS PSL CNRS INRIA) 3Catalan Institute for Research and Advanced Studies, Barcelona, Spain. |
| Pseudocode | No | The paper describes algorithms and methods using mathematical notation and text but does not include any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Code for our experiments is publicly available at github.com/facebookresearch/EGG/ as a part of the EGG framework (Kharitonov et al., 2019). |
| Open Datasets | Yes | The second game, Image Classification, employs more naturalistic data, as the agents are jointly trained to classify pairs of MNIST digits (Le Cun et al., 1998). |
| Dataset Splits | Yes | In Guess Number, we use all 2^8 possible inputs for training, early stopping and analysis. In Image Classification, we train on random image pairs from the MNIST training data, and use image pairs from the MNIST held-out set for validation. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory specifications) used for running the experiments. It focuses on the methodological aspects and results. |
| Software Dependencies | No | The paper mentions software components like 'Adam (Kingma & Ba, 2014)', 'GRU (Cho et al., 2014)', and 'Transformer (Vaswani et al., 2017) cells' but does not specify their version numbers, which is necessary for reproducibility. |
| Experiment Setup | Yes | We report hyperparameter grids in Supplementary. ... In our experiments, we found that high values of λs (the parameter controlling Sender’s entropy regularization) prevent communication success; on the other hand, a small non-zero λs is crucial for successful training. ... In Figure 1b, we split the results obtained with Gumbel Softmax by relaxation temperature. |