Semantic Representations of Speech Production in Intracranial EEG

Christian Herff; Maxime Verwoert; Joaquín Amigó-Vega; Maarten C. Ottenhoff

doi:10.1109/SMC53992.2023.10394550

Semantic Representations of Speech Production in Intracranial EEG

Christian Herff, Maxime Verwoert, Joaquín Amigó-Vega, Maarten C. Ottenhoff

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

Abstract

Speech neurprostheses have the potential to provide severely paralyzed patients with a means of communication. To enable the best possible decoding of speech processes from neural data, it is important to chose a representation of speech that is both meaningful to the decoding process and represented well in the neural recordings. Previously, acoustic, articulatory, and textual representations of speech have been decoded from neural recordings. Semantic representations of speech could add additional information about the content of the produced speech. In this study, we show that semantic embeddings for individual words, as extracted by a word2vec-model, can be used to reconstruct neural activity during speech production across wide-spread cortical and subcortical areas. We elucidate the temporal dynamics of reconstruction quality and show that a slight right hemisphere preference exists. These findings could be used to add semantic information into speech neuroprostheses in the future.

Original language	English
Title of host publication	2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings
Subtitle of host publication	Improving the Quality of Life, SMC 2023 - Proceedings
Publisher	IEEE
Pages	4764-4769
Number of pages	6
ISBN (Electronic)	9798350337020
DOIs	https://doi.org/10.1109/SMC53992.2023.10394550
Publication status	Published - 1 Oct 2023
Event	2023 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2023 - Oahu, United States Duration: 1 Oct 2023 → 4 Oct 2023 https://ieeesmc2023.org/home/

Publication series

Series	IEEE International Conference on Systems, Man, and Cybernetics (SMC). Conference Proceedings
ISSN	1062-922X

Conference

Conference	2023 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2023
Abbreviated title	IEEE SMC 2023
Country/Territory	United States
City	Oahu
Period	1/10/23 → 4/10/23
Internet address	https://ieeesmc2023.org/home/

Keywords

BCI
Brain-Computer Interfaces
NLP
semantics
stereotactic EEG
word2vec

Access to Document

10.1109/SMC53992.2023.10394550

Cite this

Herff, C., Verwoert, M., Amigó-Vega, J., & Ottenhoff, M. C. (2023). Semantic Representations of Speech Production in Intracranial EEG. In 2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings: Improving the Quality of Life, SMC 2023 - Proceedings (pp. 4764-4769). IEEE. https://doi.org/10.1109/SMC53992.2023.10394550

Herff, Christian ; Verwoert, Maxime ; Amigó-Vega, Joaquín et al. / Semantic Representations of Speech Production in Intracranial EEG. 2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings: Improving the Quality of Life, SMC 2023 - Proceedings. IEEE, 2023. pp. 4764-4769 (IEEE International Conference on Systems, Man, and Cybernetics (SMC). Conference Proceedings).

@inproceedings{e38e2f8629dd421f9058912eb79fcb7b,

title = "Semantic Representations of Speech Production in Intracranial EEG",

abstract = "Speech neurprostheses have the potential to provide severely paralyzed patients with a means of communication. To enable the best possible decoding of speech processes from neural data, it is important to chose a representation of speech that is both meaningful to the decoding process and represented well in the neural recordings. Previously, acoustic, articulatory, and textual representations of speech have been decoded from neural recordings. Semantic representations of speech could add additional information about the content of the produced speech. In this study, we show that semantic embeddings for individual words, as extracted by a word2vec-model, can be used to reconstruct neural activity during speech production across wide-spread cortical and subcortical areas. We elucidate the temporal dynamics of reconstruction quality and show that a slight right hemisphere preference exists. These findings could be used to add semantic information into speech neuroprostheses in the future.",

keywords = "BCI, Brain-Computer Interfaces, NLP, semantics, stereotactic EEG, word2vec",

author = "Christian Herff and Maxime Verwoert and Joaqu{\'i}n Amig{\'o}-Vega and Ottenhoff, {Maarten C.}",

note = "Funding Information: C.H. acknowledges funding by the Dutch Research Council (NWO) through the research project {\textquoteright}Decoding Speech In SEEG (DESIS){\textquoteright} with project number VI.Veni.194.021. MV is supported by the INTENSE consortium, which has received funding from the Dutch Research Council (NWO) with Grant Number 17619. Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2023, IEEE SMC 2023 ; Conference date: 01-10-2023 Through 04-10-2023",

year = "2023",

month = oct,

day = "1",

doi = "10.1109/SMC53992.2023.10394550",

language = "English",

series = "IEEE International Conference on Systems, Man, and Cybernetics (SMC). Conference Proceedings",

publisher = "IEEE",

pages = "4764--4769",

booktitle = "2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings",

address = "United States",

url = "https://ieeesmc2023.org/home/",

}

Herff, C , Verwoert, M, Amigó-Vega, J & Ottenhoff, MC 2023, Semantic Representations of Speech Production in Intracranial EEG. in 2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings: Improving the Quality of Life, SMC 2023 - Proceedings. IEEE, IEEE International Conference on Systems, Man, and Cybernetics (SMC). Conference Proceedings, pp. 4764-4769, 2023 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2023, Oahu, Hawaii, United States, 1/10/23. https://doi.org/10.1109/SMC53992.2023.10394550

Semantic Representations of Speech Production in Intracranial EEG. / Herff, Christian ; Verwoert, Maxime; Amigó-Vega, Joaquín et al.
2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings: Improving the Quality of Life, SMC 2023 - Proceedings. IEEE, 2023. p. 4764-4769 (IEEE International Conference on Systems, Man, and Cybernetics (SMC). Conference Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - Semantic Representations of Speech Production in Intracranial EEG

AU - Herff, Christian

AU - Verwoert, Maxime

AU - Amigó-Vega, Joaquín

AU - Ottenhoff, Maarten C.

N1 - Funding Information: C.H. acknowledges funding by the Dutch Research Council (NWO) through the research project ’Decoding Speech In SEEG (DESIS)’ with project number VI.Veni.194.021. MV is supported by the INTENSE consortium, which has received funding from the Dutch Research Council (NWO) with Grant Number 17619. Publisher Copyright: © 2023 IEEE.

PY - 2023/10/1

Y1 - 2023/10/1

N2 - Speech neurprostheses have the potential to provide severely paralyzed patients with a means of communication. To enable the best possible decoding of speech processes from neural data, it is important to chose a representation of speech that is both meaningful to the decoding process and represented well in the neural recordings. Previously, acoustic, articulatory, and textual representations of speech have been decoded from neural recordings. Semantic representations of speech could add additional information about the content of the produced speech. In this study, we show that semantic embeddings for individual words, as extracted by a word2vec-model, can be used to reconstruct neural activity during speech production across wide-spread cortical and subcortical areas. We elucidate the temporal dynamics of reconstruction quality and show that a slight right hemisphere preference exists. These findings could be used to add semantic information into speech neuroprostheses in the future.

AB - Speech neurprostheses have the potential to provide severely paralyzed patients with a means of communication. To enable the best possible decoding of speech processes from neural data, it is important to chose a representation of speech that is both meaningful to the decoding process and represented well in the neural recordings. Previously, acoustic, articulatory, and textual representations of speech have been decoded from neural recordings. Semantic representations of speech could add additional information about the content of the produced speech. In this study, we show that semantic embeddings for individual words, as extracted by a word2vec-model, can be used to reconstruct neural activity during speech production across wide-spread cortical and subcortical areas. We elucidate the temporal dynamics of reconstruction quality and show that a slight right hemisphere preference exists. These findings could be used to add semantic information into speech neuroprostheses in the future.

KW - BCI

KW - Brain-Computer Interfaces

KW - NLP

KW - semantics

KW - stereotactic EEG

KW - word2vec

U2 - 10.1109/SMC53992.2023.10394550

DO - 10.1109/SMC53992.2023.10394550

M3 - Conference article in proceeding

T3 - IEEE International Conference on Systems, Man, and Cybernetics (SMC). Conference Proceedings

SP - 4764

EP - 4769

BT - 2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings

PB - IEEE

T2 - 2023 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2023

Y2 - 1 October 2023 through 4 October 2023

ER -

Herff C , Verwoert M, Amigó-Vega J, Ottenhoff MC. Semantic Representations of Speech Production in Intracranial EEG. In 2023 IEEE International Conference on Systems, Man, and Cybernetics: Improving the Quality of Life, SMC 2023 - Proceedings: Improving the Quality of Life, SMC 2023 - Proceedings. IEEE. 2023. p. 4764-4769. (IEEE International Conference on Systems, Man, and Cybernetics (SMC). Conference Proceedings). doi: 10.1109/SMC53992.2023.10394550