SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings
Authors: Kang Liu
ICLR 2024 | Conference PDF | Archive PDF | Plain Text | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Extensive evaluations reveal that Set CSE enhances language model semantic comprehension by approximately 30% on average. Numerous real-world applications illustrate that the well-defined Set CSE framework enables complex information retrieval tasks that cannot be achieved using existing search methods. Section 5, titled "EVALUATION", details experiments on various datasets (AG News, Financial Phrase Bank, Banking77, FMTOD, GitHub Issue, English Quotes, Reuters-21578) with performance metrics (Accuracy, F1), comparisons to baselines, and discusses hyperparameter tuning. |
| Researcher Affiliation | Academia | Kang Liu Independent Researcher kangliu@umich.edu |
| Pseudocode | Yes | The paper includes "Algorithm 1 Set CSE Operation A B1 BN C1 CM" with numbered steps outlining the process. |
| Open Source Code | No | The paper states: "Additionally, we aim to create a Set CSE application interface that enables quick sentence extraction through its straightforward syntax." This indicates future plans, not an immediate release of code for the described methodology. No specific repository link or explicit statement of code availability is provided. |
| Open Datasets | Yes | The paper explicitly states the datasets used and provides citations: "AG News Title and Description (AGT and AGD) (Zhang et al., 2015)", "Financial Phrase Bank (FPB) (Malo et al., 2014)", "Banking77 (Casanueva et al., 2020)", "Facebook Multilingual Task Oriented Dataset (FMTOD) (Schuster et al., 2018)", "Git Hub Issue (Git Hub) (Ismael, 2022)", "English Quotes (Quotes) (Eltaief, 2022)", "Reuters-21578 (Reuters) (Lewis, 1997)". These are all publicly available and properly cited. |
| Dataset Splits | Yes | The paper describes how data is split: "In each Si, randomly select nsample of sentences, denoted as Qi, and concatenate remaining sentences in all Si, denoted as U. Regard Qi s as example sets and U as the evaluation set." It also specifies hyperparameters: "The hyperparameters are selected as nsample = 20, τ = 0.05, and train epoch equals 60". |
| Hardware Specification | No | The paper does not provide specific details about the hardware used to run the experiments, such as GPU models, CPU types, or memory specifications. |
| Software Dependencies | No | The paper mentions various models and techniques like BERT, RoBERTa, SimCSE, DiffCSE, MCSE, Contriever, SGPT, TFIDF, BM25, and DPR. However, it does not specify version numbers for any underlying software dependencies or libraries (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | Yes | The paper explicitly states key hyperparameters: "The hyperparameters are selected as nsample = 20, τ = 0.05, and train epoch equals 60, which are based on fine-tuning results presented in Section 7 and Appendix C." |