Changing predictor measurement procedures affected the performance of prediction models in clinical examples

Kim Luijken*, Laure Wynants, Maarten van Smeden, Ben Van Calster, Ewout W. Steyerberg, Rolf H. H. Groenwold

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Objectives: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols.

Study Design and Setting: We examined the effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy.

Results: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from -0.70 to 1.43 (0 for good calibration), whereas the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in C-statistic and scaled Brier score between derivation and validation ranged from -0.08 to +0.08 and from -0.40 to +0.16, respectively.

Conclusion: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting. (C) 2019 The Authors. Published by Elsevier Inc.

Original languageEnglish
Pages (from-to)7-18
Number of pages12
JournalJournal of Clinical Epidemiology
Volume119
DOIs
Publication statusPublished - Mar 2020

Keywords

  • Prediction model
  • Measurement error
  • Measurement heterogeneity
  • Predictive performance
  • External validation
  • Calibration
  • LOGISTIC-REGRESSION MODELS
  • INTERNAL VALIDATION
  • EXTERNAL VALIDATION
  • RISK MODELS
  • OVARIAN
  • MUTATIONS
  • DIAGNOSIS
  • IMPACT

Cite this