Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception

Mario Senden, Thomas Emmerling, Rick van Hoof, Martin Frost, Rainer Goebel

Research output: Contribution to journalArticleAcademic

Abstract

Visual mental imagery is the quasi-perceptual experience of “seeing in the mind’s eye”. While a tight correspondence between imagery and perception in terms of subjective experience is well established, their correspondence in terms of neural representations remains insufficiently understood. In the present study, we exploit the high spatial resolution of functional magnetic resonance imaging (fMRI) at 7T, the retinotopic organization of early visual cortex, and machine-learning techniques to investigate whether visual imagery of letter shapes preserves the topographic organization of perceived shapes. Sub-millimeter resolution fMRI images were obtained from early visual cortex in six subjects performing visual imagery of four different letter shapes. Predictions of imagery voxel activation patterns based on a population receptive field-encoding model and physical letter stimuli provided first evidence in favor of detailed topographic organization. Subsequent visual field reconstructions of imagery data based on the inversion of the encoding model further showed that visual imagery preserves the geometric profile of letter shapes. These results open new avenues for decoding, as we show that a denoising autoencoder can be used to pretrain a classifier purely based on perceptual data before fine-tuning it on imagery data. Finally, we show that the autoencoder can project imagery-related voxel activations onto their perceptual counterpart allowing for visually recognizable reconstructions even at the single-trial level. The latter may eventually be utilized for the development of content-based BCI letter-speller systems.
Original languageEnglish
Pages (from-to)1167-1183
Number of pages17
JournalBrain Structure & Function
Volume224
Issue number3
DOIs
Publication statusPublished - Apr 2019

Keywords

  • SPELLING DEVICE
  • WORKING-MEMORY
  • NEURAL SYSTEMS
  • BRAIN
  • REPRESENTATIONS
  • ACTIVATION
  • MODEL
  • MOTION

Cite this

@article{2acf4903d8dd4267872b1a3c47e74d5e,
title = "Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception",
abstract = "Visual mental imagery is the quasi-perceptual experience of “seeing in the mind’s eye”. While a tight correspondence between imagery and perception in terms of subjective experience is well established, their correspondence in terms of neural representations remains insufficiently understood. In the present study, we exploit the high spatial resolution of functional magnetic resonance imaging (fMRI) at 7T, the retinotopic organization of early visual cortex, and machine-learning techniques to investigate whether visual imagery of letter shapes preserves the topographic organization of perceived shapes. Sub-millimeter resolution fMRI images were obtained from early visual cortex in six subjects performing visual imagery of four different letter shapes. Predictions of imagery voxel activation patterns based on a population receptive field-encoding model and physical letter stimuli provided first evidence in favor of detailed topographic organization. Subsequent visual field reconstructions of imagery data based on the inversion of the encoding model further showed that visual imagery preserves the geometric profile of letter shapes. These results open new avenues for decoding, as we show that a denoising autoencoder can be used to pretrain a classifier purely based on perceptual data before fine-tuning it on imagery data. Finally, we show that the autoencoder can project imagery-related voxel activations onto their perceptual counterpart allowing for visually recognizable reconstructions even at the single-trial level. The latter may eventually be utilized for the development of content-based BCI letter-speller systems.",
keywords = "SPELLING DEVICE, WORKING-MEMORY, NEURAL SYSTEMS, BRAIN, REPRESENTATIONS, ACTIVATION, MODEL, MOTION",
author = "Mario Senden and Thomas Emmerling and {van Hoof}, Rick and Martin Frost and Rainer Goebel",
year = "2019",
month = "4",
doi = "10.1007/s00429-019-01828-6",
language = "English",
volume = "224",
pages = "1167--1183",
journal = "Brain Structure & Function",
issn = "1863-2653",
publisher = "Springer",
number = "3",

}

Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception. / Senden, Mario; Emmerling, Thomas; van Hoof, Rick; Frost, Martin; Goebel, Rainer.

In: Brain Structure & Function, Vol. 224, No. 3, 04.2019, p. 1167-1183.

Research output: Contribution to journalArticleAcademic

TY - JOUR

T1 - Reconstructing imagined letters from early visual cortex reveals tight topographic correspondence between visual mental imagery and perception

AU - Senden, Mario

AU - Emmerling, Thomas

AU - van Hoof, Rick

AU - Frost, Martin

AU - Goebel, Rainer

PY - 2019/4

Y1 - 2019/4

N2 - Visual mental imagery is the quasi-perceptual experience of “seeing in the mind’s eye”. While a tight correspondence between imagery and perception in terms of subjective experience is well established, their correspondence in terms of neural representations remains insufficiently understood. In the present study, we exploit the high spatial resolution of functional magnetic resonance imaging (fMRI) at 7T, the retinotopic organization of early visual cortex, and machine-learning techniques to investigate whether visual imagery of letter shapes preserves the topographic organization of perceived shapes. Sub-millimeter resolution fMRI images were obtained from early visual cortex in six subjects performing visual imagery of four different letter shapes. Predictions of imagery voxel activation patterns based on a population receptive field-encoding model and physical letter stimuli provided first evidence in favor of detailed topographic organization. Subsequent visual field reconstructions of imagery data based on the inversion of the encoding model further showed that visual imagery preserves the geometric profile of letter shapes. These results open new avenues for decoding, as we show that a denoising autoencoder can be used to pretrain a classifier purely based on perceptual data before fine-tuning it on imagery data. Finally, we show that the autoencoder can project imagery-related voxel activations onto their perceptual counterpart allowing for visually recognizable reconstructions even at the single-trial level. The latter may eventually be utilized for the development of content-based BCI letter-speller systems.

AB - Visual mental imagery is the quasi-perceptual experience of “seeing in the mind’s eye”. While a tight correspondence between imagery and perception in terms of subjective experience is well established, their correspondence in terms of neural representations remains insufficiently understood. In the present study, we exploit the high spatial resolution of functional magnetic resonance imaging (fMRI) at 7T, the retinotopic organization of early visual cortex, and machine-learning techniques to investigate whether visual imagery of letter shapes preserves the topographic organization of perceived shapes. Sub-millimeter resolution fMRI images were obtained from early visual cortex in six subjects performing visual imagery of four different letter shapes. Predictions of imagery voxel activation patterns based on a population receptive field-encoding model and physical letter stimuli provided first evidence in favor of detailed topographic organization. Subsequent visual field reconstructions of imagery data based on the inversion of the encoding model further showed that visual imagery preserves the geometric profile of letter shapes. These results open new avenues for decoding, as we show that a denoising autoencoder can be used to pretrain a classifier purely based on perceptual data before fine-tuning it on imagery data. Finally, we show that the autoencoder can project imagery-related voxel activations onto their perceptual counterpart allowing for visually recognizable reconstructions even at the single-trial level. The latter may eventually be utilized for the development of content-based BCI letter-speller systems.

KW - SPELLING DEVICE

KW - WORKING-MEMORY

KW - NEURAL SYSTEMS

KW - BRAIN

KW - REPRESENTATIONS

KW - ACTIVATION

KW - MODEL

KW - MOTION

U2 - 10.1007/s00429-019-01828-6

DO - 10.1007/s00429-019-01828-6

M3 - Article

VL - 224

SP - 1167

EP - 1183

JO - Brain Structure & Function

JF - Brain Structure & Function

SN - 1863-2653

IS - 3

ER -