AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound

Gijs Wijngaard; Elia Formisano; Michele Esposito; Michel Dumontier

AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

Abstract

Audio-language models have shown promising results in various sound understanding tasks, yet they remain limited in their ability to reason over the fine-grained semantics of sound. In this paper, we present AUDSEMTHINKER, a model whose reasoning is structured around a framework of auditory semantics inspired by human cognition. To support this, we introduce AUDSEM, a novel dataset specifically curated for semantic descriptor reasoning in audio-language models. AUDSEM addresses the persistent challenge of data contamination in zero-shot evaluations by providing a carefully filtered collection of audio samples paired with captions generated through a robust multi-stage pipeline. Our experiments demonstrate that AUDSEMTHINKER outperforms state-of-the-art models across multiple training settings, highlighting its strength in semantic audio reasoning. Both AUDSEMTHINKER and the AUDSEM dataset are released publicly.

Original language	English
Title of host publication	39th Conference on Neural Information Processing Systems, NeurIPS 2025
Publisher	Neural Information Processing Systems Foundation
Pages	8652-8681
Number of pages	30
Volume	38
ISBN (Electronic)	9798331338275
Publication status	Published - 1 Jan 2025
Event	39th Conference on Neural Information Processing Systems, NeurIPS 2025 - San Diego, United States Duration: 2 Dec 2025 → 7 Dec 2025 https://neurips.cc/Conferences/2025

Publication series

Series	Advances in Neural Information Processing Systems
Volume	38
ISSN	1049-5258

Conference

Conference	39th Conference on Neural Information Processing Systems, NeurIPS 2025
Abbreviated title	NeurIPS 2025
Country/Territory	United States
City	San Diego
Period	2/12/25 → 7/12/25
Internet address	https://neurips.cc/Conferences/2025

Cite this

@inproceedings{9f226bee6f164e38b33259097cd80993,

title = "AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound",

abstract = "Audio-language models have shown promising results in various sound understanding tasks, yet they remain limited in their ability to reason over the fine-grained semantics of sound. In this paper, we present AUDSEMTHINKER, a model whose reasoning is structured around a framework of auditory semantics inspired by human cognition. To support this, we introduce AUDSEM, a novel dataset specifically curated for semantic descriptor reasoning in audio-language models. AUDSEM addresses the persistent challenge of data contamination in zero-shot evaluations by providing a carefully filtered collection of audio samples paired with captions generated through a robust multi-stage pipeline. Our experiments demonstrate that AUDSEMTHINKER outperforms state-of-the-art models across multiple training settings, highlighting its strength in semantic audio reasoning. Both AUDSEMTHINKER and the AUDSEM dataset are released publicly.",

author = "Gijs Wijngaard and Elia Formisano and Michele Esposito and Michel Dumontier",

note = "Funding Information: This work was supported by the Dutch Research Council (NWO 406.20.GO.030 to Prof. Elia Formisano), the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. EINF-12157, Data Science Research Infrastructure (DSRI; Maastricht University) and the Dutch Province of Limburg. We would like to thank Jopik from Filmot.com for making a dataset of subtitles available for us. We would like to thank Jenia Jitsev and the LAION team for funding this work by providing compute time on both the JUWELS Booster at J\textbackslash{}u00FClich Supercomputer Centre (JSC) and Leonardo at CINECA Datacenter. We gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this part of work by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster at J\textbackslash{}u00FClich Supercomputing Centre (JSC), with access to compute provided via LAION cooperation on foundation models and datasets research. We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Extreme Access grant EHPC-EXT-2023E02-068, provided via LAION cooperation on foundation models and datasets research. Publisher Copyright: {\textcopyright} 2025 Neural information processing systems foundation. All rights reserved.; 39th Conference on Neural Information Processing Systems, NeurIPS 2025, NeurIPS 2025 ; Conference date: 02-12-2025 Through 07-12-2025",

year = "2025",

month = jan,

day = "1",

language = "English",

volume = "38",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural Information Processing Systems Foundation",

pages = "8652--8681",

booktitle = "39th Conference on Neural Information Processing Systems, NeurIPS 2025",

address = "United States",

url = "https://neurips.cc/Conferences/2025",

}

Wijngaard, G , Formisano, E , Esposito, M & Dumontier, M 2025, AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound. in 39th Conference on Neural Information Processing Systems, NeurIPS 2025. vol. 38, Neural Information Processing Systems Foundation, Advances in Neural Information Processing Systems, vol. 38, pp. 8652-8681, 39th Conference on Neural Information Processing Systems, NeurIPS 2025, San Diego, California, United States, 2/12/25.

AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound. / Wijngaard, Gijs ; Formisano, Elia ; Esposito, Michele et al.
39th Conference on Neural Information Processing Systems, NeurIPS 2025. Vol. 38 Neural Information Processing Systems Foundation, 2025. p. 8652-8681 (Advances in Neural Information Processing Systems, Vol. 38).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - AudSemThinker

T2 - 39th Conference on Neural Information Processing Systems, NeurIPS 2025

AU - Wijngaard, Gijs

AU - Formisano, Elia

AU - Esposito, Michele

AU - Dumontier, Michel

N1 - Funding Information: This work was supported by the Dutch Research Council (NWO 406.20.GO.030 to Prof. Elia Formisano), the Dutch national e-infrastructure with the support of the SURF Cooperative using grant no. EINF-12157, Data Science Research Infrastructure (DSRI; Maastricht University) and the Dutch Province of Limburg. We would like to thank Jopik from Filmot.com for making a dataset of subtitles available for us. We would like to thank Jenia Jitsev and the LAION team for funding this work by providing compute time on both the JUWELS Booster at J\u00FClich Supercomputer Centre (JSC) and Leonardo at CINECA Datacenter. We gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-centre.eu) for funding this part of work by providing computing time through the John von Neumann Institute for Computing (NIC) on the GCS Supercomputer JUWELS Booster at J\u00FClich Supercomputing Centre (JSC), with access to compute provided via LAION cooperation on foundation models and datasets research. We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Extreme Access grant EHPC-EXT-2023E02-068, provided via LAION cooperation on foundation models and datasets research. Publisher Copyright: © 2025 Neural information processing systems foundation. All rights reserved.

PY - 2025/1/1

Y1 - 2025/1/1

N2 - Audio-language models have shown promising results in various sound understanding tasks, yet they remain limited in their ability to reason over the fine-grained semantics of sound. In this paper, we present AUDSEMTHINKER, a model whose reasoning is structured around a framework of auditory semantics inspired by human cognition. To support this, we introduce AUDSEM, a novel dataset specifically curated for semantic descriptor reasoning in audio-language models. AUDSEM addresses the persistent challenge of data contamination in zero-shot evaluations by providing a carefully filtered collection of audio samples paired with captions generated through a robust multi-stage pipeline. Our experiments demonstrate that AUDSEMTHINKER outperforms state-of-the-art models across multiple training settings, highlighting its strength in semantic audio reasoning. Both AUDSEMTHINKER and the AUDSEM dataset are released publicly.

AB - Audio-language models have shown promising results in various sound understanding tasks, yet they remain limited in their ability to reason over the fine-grained semantics of sound. In this paper, we present AUDSEMTHINKER, a model whose reasoning is structured around a framework of auditory semantics inspired by human cognition. To support this, we introduce AUDSEM, a novel dataset specifically curated for semantic descriptor reasoning in audio-language models. AUDSEM addresses the persistent challenge of data contamination in zero-shot evaluations by providing a carefully filtered collection of audio samples paired with captions generated through a robust multi-stage pipeline. Our experiments demonstrate that AUDSEMTHINKER outperforms state-of-the-art models across multiple training settings, highlighting its strength in semantic audio reasoning. Both AUDSEMTHINKER and the AUDSEM dataset are released publicly.

M3 - Conference article in proceeding

VL - 38

T3 - Advances in Neural Information Processing Systems

SP - 8652

EP - 8681

BT - 39th Conference on Neural Information Processing Systems, NeurIPS 2025

PB - Neural Information Processing Systems Foundation

Y2 - 2 December 2025 through 7 December 2025

ER -

AudSemThinker: Enhancing Audio-Language Models through Reasoning over Semantics of Sound

Abstract

Publication series

Conference

Fingerprint

Cite this