Cross-validation and permutations in MVPA: validity of permutation strategies and power of cross-validation schemes

Giancarlo Valente*, Agustin Lage Castellanos, Lars Hausfeld, Federico De Martino, Elia Formisano

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Multi-Voxel Pattern Analysis (MVPA) is a well established tool to disclose weak, distributed effects in brain activity patterns. The generalization ability is assessed by testing the learning model on new, unseen data. However, when limited data is available, the decoding success is estimated using cross-validation. There is general consensus on assessing statistical significance of cross-validated accuracy with non-parametric permutation tests. In this work we focus on the false positive control of different permutation strategies and on the statistical power of different cross-validation schemes. With simulations, we show that estimating the entire cross-validation error on each permuted dataset is the only statistically valid permutation strategy. Furthermore, using both simulations and real data from the HCP WU-Minn 3T fMRI dataset, we show that, among the different cross-validation schemes, a repeated split-half cross-validation is the most powerful, despite achieving slightly lower classification accuracy, when compared to other schemes. Our findings provide additional insights into the optimization of the experimental design for MVPA, highlighting the benefits of having many short runs.

Original languageEnglish
Article number118145
Pages (from-to)1-38
Number of pages38
JournalNeuroimage
DOIs
Publication statusE-pub ahead of print - 4 May 2021

Cite this