Transfer learning of deep neural network representations for fMRI decoding

Michele Svanera, Mattia Savardi, Sergio Benini, Alberto Signoroni, Gal Raz, Talma Hendler, Lars Muckli, Rainer Goebel, Giancarlo Valente

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

BACKGROUND: Deep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data.

NEW METHOD: In this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images.

RESULTS: The fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance.

COMPARISON WITH EXISTING METHODS: The decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone.

CONCLUSION: In this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.

Original languageEnglish
Article number108319
Number of pages13
JournalJournal of Neuroscience Methods
Volume328
Early online date1 Oct 2019
DOIs
Publication statusPublished - 1 Dec 2019

Keywords

  • BRAIN
  • Brain decoding
  • Convolutional Neural Network
  • Deep learning
  • MultiVoxel Pattern Analysis
  • NATURAL IMAGES
  • NEUROSCIENCE
  • REGRESSION
  • Transfer learning
  • fMRI

Cite this

Svanera, Michele ; Savardi, Mattia ; Benini, Sergio ; Signoroni, Alberto ; Raz, Gal ; Hendler, Talma ; Muckli, Lars ; Goebel, Rainer ; Valente, Giancarlo. / Transfer learning of deep neural network representations for fMRI decoding. In: Journal of Neuroscience Methods. 2019 ; Vol. 328.
@article{b5cf7e3bdc5e41beadee68ef500c935e,
title = "Transfer learning of deep neural network representations for fMRI decoding",
abstract = "BACKGROUND: Deep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data.NEW METHOD: In this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images.RESULTS: The fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance.COMPARISON WITH EXISTING METHODS: The decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone.CONCLUSION: In this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.",
keywords = "BRAIN, Brain decoding, Convolutional Neural Network, Deep learning, MultiVoxel Pattern Analysis, NATURAL IMAGES, NEUROSCIENCE, REGRESSION, Transfer learning, fMRI",
author = "Michele Svanera and Mattia Savardi and Sergio Benini and Alberto Signoroni and Gal Raz and Talma Hendler and Lars Muckli and Rainer Goebel and Giancarlo Valente",
note = "Copyright {\circledC} 2019 Elsevier B.V. All rights reserved.",
year = "2019",
month = "12",
day = "1",
doi = "10.1016/j.jneumeth.2019.108319",
language = "English",
volume = "328",
journal = "Journal of Neuroscience Methods",
issn = "0165-0270",
publisher = "Elsevier Science",

}

Transfer learning of deep neural network representations for fMRI decoding. / Svanera, Michele; Savardi, Mattia; Benini, Sergio; Signoroni, Alberto; Raz, Gal; Hendler, Talma; Muckli, Lars; Goebel, Rainer; Valente, Giancarlo.

In: Journal of Neuroscience Methods, Vol. 328, 108319, 01.12.2019.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Transfer learning of deep neural network representations for fMRI decoding

AU - Svanera, Michele

AU - Savardi, Mattia

AU - Benini, Sergio

AU - Signoroni, Alberto

AU - Raz, Gal

AU - Hendler, Talma

AU - Muckli, Lars

AU - Goebel, Rainer

AU - Valente, Giancarlo

N1 - Copyright © 2019 Elsevier B.V. All rights reserved.

PY - 2019/12/1

Y1 - 2019/12/1

N2 - BACKGROUND: Deep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data.NEW METHOD: In this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images.RESULTS: The fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance.COMPARISON WITH EXISTING METHODS: The decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone.CONCLUSION: In this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.

AB - BACKGROUND: Deep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g., fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data.NEW METHOD: In this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images.RESULTS: The fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance.COMPARISON WITH EXISTING METHODS: The decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone.CONCLUSION: In this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of high-dimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.

KW - BRAIN

KW - Brain decoding

KW - Convolutional Neural Network

KW - Deep learning

KW - MultiVoxel Pattern Analysis

KW - NATURAL IMAGES

KW - NEUROSCIENCE

KW - REGRESSION

KW - Transfer learning

KW - fMRI

U2 - 10.1016/j.jneumeth.2019.108319

DO - 10.1016/j.jneumeth.2019.108319

M3 - Article

C2 - 31585315

VL - 328

JO - Journal of Neuroscience Methods

JF - Journal of Neuroscience Methods

SN - 0165-0270

M1 - 108319

ER -