Speech Synthesis from Stereotactic EEG using an Electrode Shaft Dependent Multi-Input Convolutional Neural Network Approach

Miguel Angrick, Maarten Ottenhoff, Sophocles Goulis, Albert J. Colon, Louis Wagner, Dean J. Krusienski, Pieter Kubben, Tanja Schultz, Christian Herff

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademic

Abstract

Neurological disorders can lead to significant impairments in speech communication and, in severe cases, cause the complete loss of the ability to speak. Brain-Computer Interfaces have shown promise as an alternative communication modality by directly transforming neural activity of speech processes into a textual or audible representations. Previous studies investigating such speech neuroprostheses relied on electrocorticography (ECoG) or microelectrode arrays that acquire neural signals from superficial areas on the cortex. While both measurement methods have demonstrated successful speech decoding, they do not capture activity from deeper brain structures and this activity has therefore not been harnessed for speech-related BCIs. In this study, we bridge this gap by adapting a previously presented decoding pipeline for speech synthesis based on ECoG signals to implanted depth electrodes (sEEG). For this purpose, we propose a multi-input convolutional neural network that extracts speech-related activity separately for each electrode shaft and estimates spectral coefficients to reconstruct an audible waveform. We evaluate our approach on open-loop data from 5 patients who conducted a recitation task of Dutch utterances. We achieve correlations of up to 0.80 between original and reconstructed speech spectrograms, which are significantly above chance level for all patients (p < 0.001). Our results indicate that sEEG can yield similar speech decoding performance to prior ECoG studies and is a promising modality for speech BCIs.
Original languageEnglish
Title of host publication2021 43rd Annual International Conference of the IEEE Engineering in Medicine &amp; Biology Society (EMBC)
PublisherIEEE
Pages6045-6048
Number of pages4
ISBN (Electronic)978-1-7281-1179-7
ISBN (Print)978-1-7281-1180-3
DOIs
Publication statusPublished - 2021
Event43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) - Mexico (Online)
Duration: 1 Nov 20215 Nov 2021
https://embc.embs.org/2021/

Conference

Conference43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)
Abbreviated titleEMBC 2021
Period1/11/215/11/21
Internet address

Keywords

  • SPOKEN

Cite this