The geometry of integration in text classification RNNs
Authors: Kyle Aitken, Vinay Venkatesh Ramasesh, Ankush Garg, Yuan Cao, David Sussillo, Niru Maheswaranathan
ICLR 2021 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | This work addresses these questions in the context of text classification, building on earlier work studying the dynamics of binary sentiment-classification networks (Maheswaranathan et al., 2019). We study text-classification tasks beyond the binary case, exploring the dynamics of RNNs trained on both natural and synthetic datasets. |
| Researcher Affiliation | Collaboration | Kyle Aitken, , Department of Physics University of Washington Seattle, WA Vinay V. Ramasesh , Blueshift, Alphabet Mountain View, CA Ankush Garg, Google Research Mountain View, CA Yuan Cao, Google Research Mountain View, CA David Sussillo, Google Research Mountain View, CA Niru Maheswaranathan Google Research Mountain View, CA |
| Pseudocode | No | The paper describes model architectures using mathematical equations, but does not include any blocks or figures labeled 'Pseudocode' or 'Algorithm' for the overall methodology. |
| Open Source Code | No | The paper does not provide an explicit statement or link to its source code for the described methodology. |
| Open Datasets | Yes | The Yelp reviews dataset (Zhang et al., 2015) consists of Yelp reviews... The Amazon reviews dataset (Zhang et al., 2015) consists of reviews of products bought on Amazon.com... The DBPedia ontology dataset (Zhang et al., 2015) consists of titles and abstracts of Wikipedia articles... The AG s news corpus (Zhang et al., 2015) contains titles and descriptions of news articles... The Go Emotions dataset (Demszky et al., 2020) contains text from 58,000 Reddit comments... |
| Dataset Splits | Yes | The Yelp reviews dataset... Each of the five classes features 130,000 training examples and 10,000 test examples. The Amazon reviews dataset... Each of the five classes features 600,000 training examples and 130,000 test examples. The DBPedia ontology dataset... Each class contains 40,000 training examples and 5,000 testing examples. The AG s news corpus... Each category features 30,000 training examples and 1,900 testing examples. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory) used for running its experiments. |
| Software Dependencies | No | The paper mentions 'Tensor Flow TF.Text Wordpiece Tokenizer' and 'ADAM optimizer (Kingma & Ba, 2014)' but does not provide specific version numbers for these or other software components to ensure reproducibility. |
| Experiment Setup | Yes | Natural experiments use a batch size of 64 with initial learning rate η = 0.01, clipping gradients to a maximum value of 30; the learning rate decays by 0.9984 every step. Synthetic experiments use a batch size of 128, initial learning rate η = 0.1, and a gradient clip of 10; the learning rate decays by 0.9997 every step. |