Decoding Lip Movements During Continuous Speech using Electrocorticography

S. Lesaja; C. Herff; G.D. Johnson; J.J. Shih; T. Schultz; D.J. Krusienski

doi:10.1109/NER.2019.8716914

Decoding Lip Movements During Continuous Speech using Electrocorticography

S. Lesaja^*, C. Herff, G.D. Johnson, J.J. Shih, T. Schultz, D.J. Krusienski

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

63 Downloads (Pure)

Abstract

Recent work has shown that it is possible to decode aspects of continuously-spoken speech from electrocorticographic (ECoG) signals recorded on the cortical surface. The ultimate objective is to develop a speech neuroprosthetic that can provide seamless, real-time synthesis of continuous speech directly from brain activity. Instead of decoding acoustic properties or classes of speech, such a neuroprosthetic might be realized by decoding articulator movements associated with speech production, as recent work highlights a representation of articulator movement in ECoG signals. The aim of this work is to investigate the neural correlates of speech-related lip movements from video recordings. We present how characteristics of lip movement can be decoded and lip-landmark positions can be predicted.

Original language	English
Title of host publication	2019 9TH INTERNATIONAL IEEE/EMBS CONFERENCE ON NEURAL ENGINEERING (NER)
Publisher	IEEE Xplore
Pages	522-525
Number of pages	4
DOIs	https://doi.org/10.1109/NER.2019.8716914
Publication status	Published - 1 Jan 2019

Access to Document

10.1109/NER.2019.8716914

Full text Final published version, 869 KBLicence: Taverne

Cite this

@inproceedings{32f94829ebba4b9cbb823e6b69f13768,

title = "Decoding Lip Movements During Continuous Speech using Electrocorticography",

abstract = "Recent work has shown that it is possible to decode aspects of continuously-spoken speech from electrocorticographic (ECoG) signals recorded on the cortical surface. The ultimate objective is to develop a speech neuroprosthetic that can provide seamless, real-time synthesis of continuous speech directly from brain activity. Instead of decoding acoustic properties or classes of speech, such a neuroprosthetic might be realized by decoding articulator movements associated with speech production, as recent work highlights a representation of articulator movement in ECoG signals. The aim of this work is to investigate the neural correlates of speech-related lip movements from video recordings. We present how characteristics of lip movement can be decoded and lip-landmark positions can be predicted.",

author = "S. Lesaja and C. Herff and G.D. Johnson and J.J. Shih and T. Schultz and D.J. Krusienski",

year = "2019",

month = jan,

day = "1",

doi = "10.1109/NER.2019.8716914",

language = "English",

pages = "522--525",

booktitle = "2019 9TH INTERNATIONAL IEEE/EMBS CONFERENCE ON NEURAL ENGINEERING (NER)",

publisher = "IEEE Xplore",

}

TY - GEN

T1 - Decoding Lip Movements During Continuous Speech using Electrocorticography

AU - Lesaja, S.

AU - Herff, C.

AU - Johnson, G.D.

AU - Shih, J.J.

AU - Schultz, T.

AU - Krusienski, D.J.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Recent work has shown that it is possible to decode aspects of continuously-spoken speech from electrocorticographic (ECoG) signals recorded on the cortical surface. The ultimate objective is to develop a speech neuroprosthetic that can provide seamless, real-time synthesis of continuous speech directly from brain activity. Instead of decoding acoustic properties or classes of speech, such a neuroprosthetic might be realized by decoding articulator movements associated with speech production, as recent work highlights a representation of articulator movement in ECoG signals. The aim of this work is to investigate the neural correlates of speech-related lip movements from video recordings. We present how characteristics of lip movement can be decoded and lip-landmark positions can be predicted.

AB - Recent work has shown that it is possible to decode aspects of continuously-spoken speech from electrocorticographic (ECoG) signals recorded on the cortical surface. The ultimate objective is to develop a speech neuroprosthetic that can provide seamless, real-time synthesis of continuous speech directly from brain activity. Instead of decoding acoustic properties or classes of speech, such a neuroprosthetic might be realized by decoding articulator movements associated with speech production, as recent work highlights a representation of articulator movement in ECoG signals. The aim of this work is to investigate the neural correlates of speech-related lip movements from video recordings. We present how characteristics of lip movement can be decoded and lip-landmark positions can be predicted.

U2 - 10.1109/NER.2019.8716914

DO - 10.1109/NER.2019.8716914

M3 - Conference article in proceeding

SP - 522

EP - 525

BT - 2019 9TH INTERNATIONAL IEEE/EMBS CONFERENCE ON NEURAL ENGINEERING (NER)

PB - IEEE Xplore

ER -