Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The embrace of open science: An analysis of a decade of AI research and 56 800 conference papers," Under Review, 2026..
Beyond Token Probes: Hallucination Detection via Activation Tensors with ACT-ViT
Authors: Guy Bar-Shalom, Fabrizio Frasca, Yaniv Galron, Yftah Ziser, Haggai Maron
NeurIPS 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Through comprehensive experiments encompassing diverse LLMs and datasets, we demonstrate that ACT-ViT consistently outperforms traditional probing techniques while remaining extremely efficient for deployment. |
| Researcher Affiliation | Collaboration | Guy Bar-Shalom Technion EMAIL Fabrizio Frasca Technion EMAIL Yaniv Galron Technion Yftah Ziser University of Groningen, Nvidia Research Haggai Maron Technion, Nvidia Research |
| Pseudocode | Yes | see Algorithm 1, in Appendix B. Algorithm 1 Pooling |
| Open Source Code | Yes | Full code is available at https://github.com/Bar SGuy/ACT-Vi T. |
| Open Datasets | Yes | Aligning with prior work, we focus on the datasets and LLMs considered in [48, 6]. We experiment with Mistral-7B-Instruct-v0.2 [27] (Mis-7B) and Llama-3-8B-Instruct [63] (LlaMa-8B) over the generation tasks encompassed by the following datasets: Trivia QA [28] (question answering), Hotpot QA with (HQA-Wc) and without supporting context (HQA) [67] (question answering), IMDB movie review [40] (sentiment analysis), and Movies [48] (actor role retrieval). |
| Dataset Splits | Yes | For all datasets, we used a consistent split of 10,000 training samples and 10,000 test samples, unless otherwise specified. From the 10,000 training samples, 20% (i.e., 2,000) were selected in a stratified manner for validation, using a fixed random seed of 42. |
| Hardware Specification | Yes | Our experiments were conducted using the PyTorch [50] framework (License: BSD), using a single NVIDIA L-40 GPU for all experiments. |
| Software Dependencies | Yes | Our experiments were conducted using the PyTorch [50] framework (License: BSD), using a single NVIDIA L-40 GPU for all experiments. We use a fixed batch size of 128 for all experiments, other than the ones with ACT-Vi T(s), ACT-MLP(s), where we used a batch size of 64. We used 4 heads in the transformer part of Vi T for all experiments. Hyperparameter tuning was performed utilizing the Weight and Biases framework [9] see Appendix A.1. |
| Experiment Setup | Yes | We use a fixed batch size of 128 for all experiments, other than the ones with ACT-Vi T(s), ACT-MLP(s), where we used a batch size of 64. We used 4 heads in the transformer part of Vi T for all experiments. Hyperparameter tuning was performed utilizing the Weight and Biases framework [9] see Appendix A.1. Optimizer and Schedulers. For all datasets, we use the Adam W optimizer [39] in combination with a cosine learning rate scheduler, incorporating a warm-up phase over the first 10% of training epochs. |