Skip to main navigation Skip to search Skip to main content

Language prediction in multimodal contexts: The contribution of iconic gestures to anticipatory sentence comprehension

  • Florian Hintz*
  • , Alexandra K. Emmendorfer
  • , Antje Strauss
  • , Yung Han Khoe
  • , Lydia Moonen
  • , Judith Holler
  • *Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

Evidence shows listeners predict upcoming words, and iconic gestures in face-to-face dialogue often precede their spoken referents. A recent framework proposes that early-onset gestures can pre-activate the meaning of upcoming nouns. This pre-registered EEG study tested this proposal. 81 Dutch adults watched videos of an actress producing discourses ending with a target noun. The noun was either predictable or not from the discourse. Each noun was paired with an iconic gesture whose stroke began either 520 msec (earlier) or 130 msec (later) before the noun. Discourse predictability produced the hypothesized post-target N400 reduction for predictable versus non-predictable items (H1). In the pre-target window, earlier gestures in non-predictable discourses elicited more negative amplitudes than in predictable discourses (H2a), suggesting that earlier gestures modulated neural responses when predictability was low. No comparable interaction was found for the opposite contrast (early-predictable vs late-predictable & late-non-predictable; H2c). In the post-target window, earlier gestures yielded a modest N400-like positivity relative to late gestures (H2e), but the interaction between gesture timing and discourse predictability (H2f) was not supported. A follow-up indicated that, in non-predictable discourses, later gestures produced a larger N400 than earlier gestures. Thus, the data confirm H1 and support H2a (and partially H2e), while H2c and H2f were not supported, suggesting that when speakers produce monologic discourses, earlier iconic gestures can aid anticipatory processing when linguistic predictions are weak, but their effect is limited and context-dependent. Together with the exploratory follow-up, the findings highlight a nuanced role of the influence of gesture timing on comprehension. (c) 2026 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Original languageEnglish
Pages (from-to)415-440
Number of pages26
JournalCortex
Volume205
Early online date29 Sept 2026
DOIs
Publication statusE-pub ahead of print - 29 Sept 2026

Keywords

  • Multimodal communication
  • Language comprehension
  • Prediction
  • Gesture-speech integration
  • Iconic co-speech gestures
  • NEURAL INTEGRATION
  • DISCOURSE COMPREHENSION
  • SEMANTIC INFORMATION
  • PROACTIVE BRAIN
  • HAND GESTURES
  • SPEECH
  • ACTIVATION
  • MEMORY
  • WORDS
  • KNOWLEDGE

Fingerprint

Dive into the research topics of 'Language prediction in multimodal contexts: The contribution of iconic gestures to anticipatory sentence comprehension'. Together they form a unique fingerprint.

Cite this