TY - JOUR
T1 - Challenges and caveats of a multi-center retrospective radiomics study
T2 - an example of early treatment response assessment for NSCLC patients using FDG-PET/CT radiomics
AU - van Timmeren, Janna E.
AU - Carvalho, Sara
AU - Leijenaar, Ralph T. H.
AU - Troost, Esther G. C.
AU - van Elmpt, Wouter
AU - de Ruysscher, Dirk
AU - Muratet, Jean-Pierre
AU - Denis, Fabrice
AU - Schimek-Jasch, Tanj A.
AU - Nestle, Ursula
AU - Jochems, Arthur
AU - Woodruff, Henry C.
AU - Oberije, Cary
AU - Lambin, Philippe
PY - 2019/6/3
Y1 - 2019/6/3
N2 - BackgroundPrognostic models based on individual patient characteristics can improve treatment decisions and outcome in the future. In many (radiomic) studies, small size and heterogeneity of datasets is a challenge that often limits performance and potential clinical applicability of these models. The current study is example of a retrospective multi-centric study with challenges and caveats. To highlight common issues and emphasize potential pitfalls, we aimed for an extensive analysis of these multi-center pre-treatment datasets, with an additional 18 F-fluorodeoxyglucose (FDG) positron emission tomography/computed tomography (PET/CT) scan acquired during treatment.MethodsThe dataset consisted of 138 stage II-IV non-small cell lung cancer (NSCLC) patients from four different cohorts acquired from three different institutes. The differences between the cohorts were compared in terms of clinical characteristics and using the so-called 'cohort differences model' approach. Moreover, the potential prognostic performances for overall survival of radiomic features extracted from CT or FDG-PET, or relative or absolute differences between the scans at the two time points, were assessed using the LASSO regression method. Furthermore, the performances of five different classifiers were evaluated for all image sets.ResultsThe individual cohorts substantially differed in terms of patient characteristics. Moreover, the cohort differences model indicated statistically significant differences between the cohorts. Neither LASSO nor any of the tested classifiers resulted in a clinical relevant prognostic model that could be validated on the available datasets.ConclusionThe results imply that the study might have been influenced by a limited sample size, heterogeneous patient characteristics, and inconsistent imaging parameters. No prognostic performance of FDG-PET or CT based radiomics models can be reported. This study highlights the necessity of extensive evaluations of cohorts and of validation datasets, especially in retrospective multi-centric datasets.
AB - BackgroundPrognostic models based on individual patient characteristics can improve treatment decisions and outcome in the future. In many (radiomic) studies, small size and heterogeneity of datasets is a challenge that often limits performance and potential clinical applicability of these models. The current study is example of a retrospective multi-centric study with challenges and caveats. To highlight common issues and emphasize potential pitfalls, we aimed for an extensive analysis of these multi-center pre-treatment datasets, with an additional 18 F-fluorodeoxyglucose (FDG) positron emission tomography/computed tomography (PET/CT) scan acquired during treatment.MethodsThe dataset consisted of 138 stage II-IV non-small cell lung cancer (NSCLC) patients from four different cohorts acquired from three different institutes. The differences between the cohorts were compared in terms of clinical characteristics and using the so-called 'cohort differences model' approach. Moreover, the potential prognostic performances for overall survival of radiomic features extracted from CT or FDG-PET, or relative or absolute differences between the scans at the two time points, were assessed using the LASSO regression method. Furthermore, the performances of five different classifiers were evaluated for all image sets.ResultsThe individual cohorts substantially differed in terms of patient characteristics. Moreover, the cohort differences model indicated statistically significant differences between the cohorts. Neither LASSO nor any of the tested classifiers resulted in a clinical relevant prognostic model that could be validated on the available datasets.ConclusionThe results imply that the study might have been influenced by a limited sample size, heterogeneous patient characteristics, and inconsistent imaging parameters. No prognostic performance of FDG-PET or CT based radiomics models can be reported. This study highlights the necessity of extensive evaluations of cohorts and of validation datasets, especially in retrospective multi-centric datasets.
KW - CELL LUNG-CANCER
KW - PROGNOSTIC VALUE
KW - FEATURES
KW - RECONSTRUCTION
KW - SUV
KW - DISCRETIZATION
KW - VARIABILITY
KW - PREDICTION
KW - SURVIVAL
KW - IMPACT
U2 - 10.1371/journal.pone.0217536
DO - 10.1371/journal.pone.0217536
M3 - Article
C2 - 31158263
VL - 14
JO - PLOS ONE
JF - PLOS ONE
SN - 1932-6203
IS - 6
M1 - 0217536
ER -