RMIB: Representation Matching Information Bottleneck for Matching Text Representations
Authors: Haihui Pan, Zhifang Liao, Wenrui Xie, Kun Han
ICML 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | On four text matching models and five text matching datasets, we verify that RMIB can improve the performance of asymmetrical domains text matching. Our experimental code is available at https://github.com/chenxingphh/rmib. We provide both theoretical foundations and empirical evidence to verify the effectiveness of RMIB. |
| Researcher Affiliation | Collaboration | 1Central South University, Changsha, China 2Cheetah Mobile Inc., Beijing, China 3Datawhale Org., Hangzhou, China. |
| Pseudocode | No | Not found. The paper includes Figure 1 which illustrates the optimization objective, but it is not a pseudocode or algorithm block. |
| Open Source Code | Yes | Our experimental code is available at https://github.com/chenxingphh/rmib. |
| Open Datasets | Yes | SICK (Sentences Involving Compositional Knowledge) (Marelli et al., 2014) is a dataset for compositional distributional semantics, which consists of 9.8k pairs of sentences. Sci Tail (Science Entailment) (Khot et al., 2018) is an entailment dataset created from multiple-choice science exams and web sentences. Wiki QA (Yang et al., 2015) is a retrieval-based question answering dataset based on Wikipedia. SNLI (Stanford Natural Language Inference) (Bowman et al., 2015) is a benchmark dataset for natural language inference which contains 570k human annotated sentence pairs. Quora Question Pair is a dataset for paraphrase identification. MRPC(Microsoft Research Paraphrase Corpus) (Dolan & Brockett, 2005) is a corpus consisting of 5,801 sentence pairs collected from newswire articles. The RTE(Recognizing Textual Entailment) (Wang et al., 2018) datasets come from a series of textual entailment challenges. The WNLI (Wang et al., 2018) contains pairs of sentences, and the task is to determine whether the second sentence is an entailment of the first one or not. |
| Dataset Splits | Yes | The dataset contains 20.4k training pairs, 2.7k development pairs, and 6.2k test pairs. The data preprocessing of SNLI, Quora Question Pair, Sci Tail and Wiki QA is consistent with RE2 (Yang et al., 2019). |
| Hardware Specification | No | Not found. The paper does not specify any particular CPU, GPU, or other hardware used for experiments. |
| Software Dependencies | No | Not found. The paper mentions tools and models like GloVe, RE2, ESIM, BERT, SBERT, and Adam optimizer, but does not provide specific version numbers for any software dependencies. |
| Experiment Setup | Yes | For BERT (Kenton & Toutanova, 2019) and SBERT (Reimers & Gurevych, 2019), we use the max pooling to obtain the text representations, the learning rate is 2e-5 and the epoch is 6. All experiments use Adam (Kingma & Ba, 2014) as the optimizer and the hyper-parameters search range for α1, α2 and α3 in RMIB is {0.01, 0.02, 0.03}. To ensure the reproducibility of the results, all experiments use the same seed. |