TY - JOUR
T1 - METhodological RadiomICs Score (METRICS)
T2 - a quality scoring tool for radiomics research endorsed by EuSoMII
AU - Kocak, Burak
AU - Akinci D'Antonoli, Tugba
AU - Mercaldo, Nathaniel
AU - Alberich-Bayarri, Angel
AU - Baessler, Bettina
AU - Ambrosini, Ilaria
AU - Andreychenko, Anna E.
AU - Bakas, Spyridon
AU - Beets-Tan, Regina G. H.
AU - Bressem, Keno
AU - Buvat, Irene
AU - Cannella, Roberto
AU - Cappellini, Luca Alessandro
AU - Cavallo, Armando Ugo
AU - Chepelev, Leonid L.
AU - Chu, Linda Chi Hang
AU - Demircioglu, Aydin
AU - deSouza, Nandita M.
AU - Dietzel, Matthias
AU - Fanni, Salvatore Claudio
AU - Fedorov, Andrey
AU - Fournier, Laure S.
AU - Giannini, Valentina
AU - Girometti, Rossano
AU - Groot Lipman, Kevin B. W.
AU - Kalarakis, Georgios
AU - Kelly, Brendan S.
AU - Klontzas, Michail E.
AU - Koh, Dow-Mu
AU - Kotter, Elmar
AU - Lee, Ho Yun
AU - Maas, Mario
AU - Marti-Bonmati, Luis
AU - Muller, Henning
AU - Obuchowski, Nancy
AU - Orlhac, Fanny
AU - Papanikolaou, Nikolaos
AU - Petrash, Ekaterina
AU - Pfaehler, Elisabeth
AU - Pinto dos Santos, Daniel
AU - Ponsiglione, Andrea
AU - Sabater, Sebastia
AU - Sardanelli, Francesco
AU - Seeboeck, Philipp
AU - Sijtsema, Nanna M.
AU - Stanzione, Arnaldo
AU - Traverso, Alberto
AU - Ugga, Lorenzo
AU - Vallieres, Martin
AU - van Dijk, Lisanne V.
AU - Et al.
PY - 2024/1/17
Y1 - 2024/1/17
N2 - PurposeTo propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.MethodsWe conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.ResultIn total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.ConclusionIn this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.Critical relevance statementA quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).
AB - PurposeTo propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.MethodsWe conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.ResultIn total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.ConclusionIn this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers.Critical relevance statementA quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning.Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).Key points center dot A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol.center dot The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time.center dot METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines.center dot A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics).
KW - Radiomics
KW - Deep learning
KW - Artificial intelligence
KW - Machine learning
KW - Guideline
U2 - 10.1186/s13244-023-01572-w
DO - 10.1186/s13244-023-01572-w
M3 - Article
SN - 1869-4101
VL - 15
JO - Insights into Imaging
JF - Insights into Imaging
IS - 1
M1 - 8
ER -