TY - JOUR
T1 - Multivariable prediction of functional outcome after first-episode psychosis
T2 - a crossover validation approach in EUFEST and PSYSCAN
AU - Slot, Margot I E
AU - Urquijo Castro, Maria F
AU - Winter-van Rossum, Inge
AU - van Hell, Hendrika H
AU - Dwyer, Dominic
AU - Dazzan, Paola
AU - Maat, Arija
AU - De Haan, Lieuwe
AU - Crespo-Facorro, Benedicto
AU - Glenthøj, Birte Y
AU - Lawrie, Stephen M
AU - McDonald, Colm
AU - Gruber, Oliver
AU - van Amelsvoort, Thérèse
AU - Arango, Celso
AU - Kircher, Tilo
AU - Nelson, Barnaby
AU - Galderisi, Silvana
AU - Weiser, Mark
AU - Sachs, Gabriele
AU - Kirschner, Matthias
AU - Fleischhacker, W Wolfgang
AU - McGuire, Philip
AU - Koutsouleris, Nikolaos
AU - Kahn, René S
AU - PSYSCAN Consortium
AU - Marcelis, Machteld
AU - Vingerhoets, Claudia
PY - 2024/10/7
Y1 - 2024/10/7
N2 - Several multivariate prognostic models have been published to predict outcomes in patients with first episode psychosis (FEP), but it remains unclear whether those predictions generalize to independent populations. Using a subset of demographic and clinical baseline predictors, we aimed to develop and externally validate different models predicting functional outcome after a FEP in the context of a schizophrenia-spectrum disorder (FES), based on a previously published cross-validation and machine learning pipeline. A crossover validation approach was adopted in two large, international cohorts (EUFEST, n = 338, and the PSYSCAN FES cohort, n = 226). Scores on the Global Assessment of Functioning scale (GAF) at 12 month follow-up were dichotomized to differentiate between poor (GAF current < 65) and good outcome (GAF current = 65). Pooled non-linear support vector machine (SVM) classifiers trained on the separate cohorts identified patients with a poor outcome with cross-validated balanced accuracies (BAC) of 65-66%, but BAC dropped substantially when the models were applied to patients from a different FES cohort (BAC = 50-56%). A leave-site-out analysis on the merged sample yielded better performance (BAC = 72%), highlighting the effect of combining data from different study designs to overcome calibration issues and improve model transportability. In conclusion, our results indicate that validation of prediction models in an independent sample is essential in assessing the true value of the model. Future external validation studies, as well as attempts to harmonize data collection across studies, are recommended.
AB - Several multivariate prognostic models have been published to predict outcomes in patients with first episode psychosis (FEP), but it remains unclear whether those predictions generalize to independent populations. Using a subset of demographic and clinical baseline predictors, we aimed to develop and externally validate different models predicting functional outcome after a FEP in the context of a schizophrenia-spectrum disorder (FES), based on a previously published cross-validation and machine learning pipeline. A crossover validation approach was adopted in two large, international cohorts (EUFEST, n = 338, and the PSYSCAN FES cohort, n = 226). Scores on the Global Assessment of Functioning scale (GAF) at 12 month follow-up were dichotomized to differentiate between poor (GAF current < 65) and good outcome (GAF current = 65). Pooled non-linear support vector machine (SVM) classifiers trained on the separate cohorts identified patients with a poor outcome with cross-validated balanced accuracies (BAC) of 65-66%, but BAC dropped substantially when the models were applied to patients from a different FES cohort (BAC = 50-56%). A leave-site-out analysis on the merged sample yielded better performance (BAC = 72%), highlighting the effect of combining data from different study designs to overcome calibration issues and improve model transportability. In conclusion, our results indicate that validation of prediction models in an independent sample is essential in assessing the true value of the model. Future external validation studies, as well as attempts to harmonize data collection across studies, are recommended.
U2 - 10.1038/s41537-024-00505-w
DO - 10.1038/s41537-024-00505-w
M3 - Article
SN - 2754-6993
VL - 10
JO - Schizophrenia
JF - Schizophrenia
IS - 1
M1 - 89
ER -