Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in [1].
Audio Entailment: Assessing Deductive Reasoning for Audio Understanding
Authors: Soham Deshmukh, Shuo Han, Hazim Bukhari, Benjamin Elizalde, Hannes Gamper, Rita Singh, Bhiksha Raj
AAAI 2025 | Venue PDF | LLM Run Details
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We introduce the novel task of Audio Entailment to evaluate an ALM s deductive reasoning ability. This task assesses whether a text description (hypothesis) of audio content can be deduced from an audio recording (premise)... We create two datasets for this task... We benchmark state-of-the-art ALMs and find deficiencies in logical reasoning with both zero-shot and linear probe evaluations. |
| Researcher Affiliation | Collaboration | 1Carnegie Mellon University 2Microsoft 3Microsoft Research EMAIL |
| Pseudocode | No | No specific pseudocode or algorithm blocks are present in the provided text. The methods are described narratively. |
| Open Source Code | No | The paper states "Datasets https://github.com/microsoft/Audio Entailment" which only refers to datasets, not code. In section 4.2, it mentions "The datasets, models, and code will be publicly released.", which is a future promise rather than a concrete provision of code at the time of publication or accessible via the provided link. |
| Open Datasets | Yes | Datasets https://github.com/microsoft/Audio Entailment. We created two datasets, ACE and CLE, where Hypotheses were first generated by GPT-4 and then verified and corrected by human annotators. This two-step process enhances the quality of the datasets, which will be publicly released. |
| Dataset Splits | Yes | Data Split Dur. H Median Max Vocab. [hrs] [chars] [chars] [words] CLE train 23.98 3839 68 195 4678 CLE val 6.56 1045 69 208 2828 CLE test 6.50 1045 67 192 2759 ACE test 2.63 4785 57 207 3901 |
| Hardware Specification | Yes | We conduct experiments using ACE and CLE in Section 5 on 80GB A100 GPU. |
| Software Dependencies | No | No specific ancillary software details with version numbers are provided in the paper. |
| Experiment Setup | No | The paper describes the general experimental setup such as framing the task as a 3-way classification, using non-overlapping similarity thresholds, and linear-probe setups where a classifier is trained. However, specific hyperparameters like learning rates, batch sizes, or optimizer settings for training the classifier, or specific LLM prompting parameters (e.g., temperature) are not explicitly mentioned in the main text, with some details being deferred to an appendix. |