Unraveling Uncertainty: The Impact of Biological and Analytical Variation on the Prediction Uncertainty of Categorical Prediction Models

Remy J H Martens*, William P T M van Doorn, Mathie P G Leers, Steven J R Meex, Floris Helmich

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

BACKGROUND: Interest in prediction models, including machine learning (ML) models, based on laboratory data has increased tremendously. Uncertainty in laboratory measurements and predictions based on such data are inherently intertwined. This study developed a framework for assessing the impact of biological and analytical variation on the prediction uncertainty of categorical prediction models. METHODS: Practical application was demonstrated for the prediction of renal function loss (Chronic Kidney Disease Epidemiology Collaboration [CKD-EPI] equation) and 31-day mortality (advanced ML model) in 6360 emergency department patients. Model outcome was calculated in 100 000 simulations of variation in laboratory parameters. Subsequently, the percentage of discordant predictions was calculated with the original prediction as reference. Simulations were repeated assuming increasing levels of analytical variation. RESULTS: For the ML model, area under the receiver operating characteristic curve (AUROC), sensitivity, and specificity were 0.90, 0.44, and 0.96, respectively. At base analytical variation, the median [2.5th-97.5th percentiles] percentage of discordant predictions was 0% [0%-28.8%]. In addition, 7.2% of patients had >5% discordant predictions. At 6× base analytical variation, the median [2.5th-97.5th percentiles] percentage of discordant predictions was 0% [0%-38.8%]. In addition, 11.7% of patients had >5% discordant predictions. However, the impact of analytical variation was limited compared with biological variation. AUROC, sensitivity, and specificity were not affected by variation in laboratory parameters. CONCLUSIONS: The impact of biological and analytical variation on the prediction uncertainty of categorical prediction models, including ML models, can be estimated by the occurrence of discordant predictions in a simulation model. Nevertheless, discordant predictions at the individual level do not necessarily affect model performance at the population level.
Original languageEnglish
Article numberjfae115
Number of pages13
JournalJournal of Applied Laboratory Medicine
DOIs
Publication statusE-pub ahead of print - 5 Nov 2024

Fingerprint

Dive into the research topics of 'Unraveling Uncertainty: The Impact of Biological and Analytical Variation on the Prediction Uncertainty of Categorical Prediction Models'. Together they form a unique fingerprint.

Cite this