Audio-tactile enhancement of cortical speech-envelope tracking

Lars Riecke, Sophia Snipes, Sander van Bree, Amanda Kaas, Lars Hausfeld

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Viewing a speaker's lip movements can improve the brain's ability to 'track' the amplitude envelope of the auditory speech signal and facilitate intelligibility. Whether such neurobehavioral benefits can also arise from tactually sensing the speech envelope on the skin is unclear. We hypothesized that tactile speech envelopes can improve neural tracking of auditory speech and thereby facilitate intelligibility. To test this, we applied continuous auditory speech and vibrotactile speech-envelope-shaped stimulation at various asynchronies to the ears and index fingers of normally-hearing human listeners while simultaneously assessing speech-recognition performance and cortical speech-envelope tracking with electroencephalography. Results indicate that tactile speech-shaped envelopes improve the cortical tracking, but not intelligibility, of degraded auditory speech. The cortical speech-tracking benefit occurs for tactile input leading the auditory input by 100 m s or less, emerges in the EEG during an early time window (∼0-150 m s), and in particular involves cortical activity in the delta (1-4 Hz) range. These characteristics hint at a predictive mechanism for multisensory integration of complex slow time-varying inputs that might play a role in tactile speech communication.

Original languageEnglish
Article number116134
Number of pages9
JournalNeuroimage
Volume202
Early online date27 Aug 2019
DOIs
Publication statusPublished - 15 Nov 2019

Keywords

  • Electroencephalography
  • Multisensory
  • Speech envelope
  • Speech tracking
  • Tactile
  • AUDIOVISUAL SPEECH
  • NEURONAL OSCILLATIONS
  • AUDITORY-PERCEPTION
  • TEMPORAL ENVELOPE
  • EVOKED-POTENTIALS
  • INTEGRATION
  • PHASE
  • RESPONSES
  • CORTEX
  • INFORMATION

Cite this

@article{26d951ea2cd942479c22689fa5b7a2d8,
title = "Audio-tactile enhancement of cortical speech-envelope tracking",
abstract = "Viewing a speaker's lip movements can improve the brain's ability to 'track' the amplitude envelope of the auditory speech signal and facilitate intelligibility. Whether such neurobehavioral benefits can also arise from tactually sensing the speech envelope on the skin is unclear. We hypothesized that tactile speech envelopes can improve neural tracking of auditory speech and thereby facilitate intelligibility. To test this, we applied continuous auditory speech and vibrotactile speech-envelope-shaped stimulation at various asynchronies to the ears and index fingers of normally-hearing human listeners while simultaneously assessing speech-recognition performance and cortical speech-envelope tracking with electroencephalography. Results indicate that tactile speech-shaped envelopes improve the cortical tracking, but not intelligibility, of degraded auditory speech. The cortical speech-tracking benefit occurs for tactile input leading the auditory input by 100 m s or less, emerges in the EEG during an early time window (∼0-150 m s), and in particular involves cortical activity in the delta (1-4 Hz) range. These characteristics hint at a predictive mechanism for multisensory integration of complex slow time-varying inputs that might play a role in tactile speech communication.",
keywords = "Electroencephalography, Multisensory, Speech envelope, Speech tracking, Tactile, AUDIOVISUAL SPEECH, NEURONAL OSCILLATIONS, AUDITORY-PERCEPTION, TEMPORAL ENVELOPE, EVOKED-POTENTIALS, INTEGRATION, PHASE, RESPONSES, CORTEX, INFORMATION",
author = "Lars Riecke and Sophia Snipes and {van Bree}, Sander and Amanda Kaas and Lars Hausfeld",
note = "Copyright {\circledC} 2019. Published by Elsevier Inc.",
year = "2019",
month = "11",
day = "15",
doi = "10.1016/j.neuroimage.2019.116134",
language = "English",
volume = "202",
journal = "Neuroimage",
issn = "1053-8119",
publisher = "Elsevier Science",

}

Audio-tactile enhancement of cortical speech-envelope tracking. / Riecke, Lars; Snipes, Sophia; van Bree, Sander; Kaas, Amanda; Hausfeld, Lars.

In: Neuroimage, Vol. 202, 116134, 15.11.2019.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Audio-tactile enhancement of cortical speech-envelope tracking

AU - Riecke, Lars

AU - Snipes, Sophia

AU - van Bree, Sander

AU - Kaas, Amanda

AU - Hausfeld, Lars

N1 - Copyright © 2019. Published by Elsevier Inc.

PY - 2019/11/15

Y1 - 2019/11/15

N2 - Viewing a speaker's lip movements can improve the brain's ability to 'track' the amplitude envelope of the auditory speech signal and facilitate intelligibility. Whether such neurobehavioral benefits can also arise from tactually sensing the speech envelope on the skin is unclear. We hypothesized that tactile speech envelopes can improve neural tracking of auditory speech and thereby facilitate intelligibility. To test this, we applied continuous auditory speech and vibrotactile speech-envelope-shaped stimulation at various asynchronies to the ears and index fingers of normally-hearing human listeners while simultaneously assessing speech-recognition performance and cortical speech-envelope tracking with electroencephalography. Results indicate that tactile speech-shaped envelopes improve the cortical tracking, but not intelligibility, of degraded auditory speech. The cortical speech-tracking benefit occurs for tactile input leading the auditory input by 100 m s or less, emerges in the EEG during an early time window (∼0-150 m s), and in particular involves cortical activity in the delta (1-4 Hz) range. These characteristics hint at a predictive mechanism for multisensory integration of complex slow time-varying inputs that might play a role in tactile speech communication.

AB - Viewing a speaker's lip movements can improve the brain's ability to 'track' the amplitude envelope of the auditory speech signal and facilitate intelligibility. Whether such neurobehavioral benefits can also arise from tactually sensing the speech envelope on the skin is unclear. We hypothesized that tactile speech envelopes can improve neural tracking of auditory speech and thereby facilitate intelligibility. To test this, we applied continuous auditory speech and vibrotactile speech-envelope-shaped stimulation at various asynchronies to the ears and index fingers of normally-hearing human listeners while simultaneously assessing speech-recognition performance and cortical speech-envelope tracking with electroencephalography. Results indicate that tactile speech-shaped envelopes improve the cortical tracking, but not intelligibility, of degraded auditory speech. The cortical speech-tracking benefit occurs for tactile input leading the auditory input by 100 m s or less, emerges in the EEG during an early time window (∼0-150 m s), and in particular involves cortical activity in the delta (1-4 Hz) range. These characteristics hint at a predictive mechanism for multisensory integration of complex slow time-varying inputs that might play a role in tactile speech communication.

KW - Electroencephalography

KW - Multisensory

KW - Speech envelope

KW - Speech tracking

KW - Tactile

KW - AUDIOVISUAL SPEECH

KW - NEURONAL OSCILLATIONS

KW - AUDITORY-PERCEPTION

KW - TEMPORAL ENVELOPE

KW - EVOKED-POTENTIALS

KW - INTEGRATION

KW - PHASE

KW - RESPONSES

KW - CORTEX

KW - INFORMATION

U2 - 10.1016/j.neuroimage.2019.116134

DO - 10.1016/j.neuroimage.2019.116134

M3 - Article

C2 - 31470124

VL - 202

JO - Neuroimage

JF - Neuroimage

SN - 1053-8119

M1 - 116134

ER -