Prospective validation of pathologic complete response models in rectal cancer: Transferability and reproducibility

Johan van Soest, Elisa Meldolesi, Ruud van Stiphout, Roberto Gatta, Andrea Damiani, Vincenzo Valentini, Philippe Lambin, Andre Dekker

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Purpose: Multiple models have been developed to predict pathologic complete response (pCR) in locally advanced rectal cancer patients. Unfortunately, validation of these models normally omit the implications of cohort differences on prediction model performance. In this work, we will perform a prospective validation of three pCR models, including information whether this validation will target transferability or reproducibility (cohort differences) of the given models. 
Methods: We applied a novel methodology, the cohort differences model, to predict whether a patient belongs to the training or to the validation cohort. If the cohort differences model performs well, it would suggest a large difference in cohort characteristics meaning we would validate the transferability of the model rather than reproducibility. We tested our method in a prospective validation of three existing models for pCR prediction in 154 patients. 
Results: Our results showed a large difference between training and validation cohort for one of the three tested models [Area under the Receiver Operating Curve (AUC) cohort differences model: 0.85], signaling the validation leans towards transferability. Two out of three models had a lower AUC for validation (0.66 and 0.58), one model showed a higher AUC in the validation cohort (0.70).
Discussion: We have successfully applied a new methodology in the validation of three prediction models, which allows us to indicate if a validation targeted transferability (large differences between training/validation cohort) or reproducibility (small cohort differences).
Original languageEnglish
JournalMedical Physics
Volume44
Issue number9
DOIs
Publication statusPublished - 2017

Cite this

van Soest, Johan ; Meldolesi, Elisa ; van Stiphout, Ruud ; Gatta, Roberto ; Damiani, Andrea ; Valentini, Vincenzo ; Lambin, Philippe ; Dekker, Andre. / Prospective validation of pathologic complete response models in rectal cancer : Transferability and reproducibility. In: Medical Physics. 2017 ; Vol. 44, No. 9.
@article{59c3089a45e34a59a3d80833d91cdb0a,
title = "Prospective validation of pathologic complete response models in rectal cancer: Transferability and reproducibility",
abstract = "Purpose: Multiple models have been developed to predict pathologic complete response (pCR) in locally advanced rectal cancer patients. Unfortunately, validation of these models normally omit the implications of cohort differences on prediction model performance. In this work, we will perform a prospective validation of three pCR models, including information whether this validation will target transferability or reproducibility (cohort differences) of the given models. Methods: We applied a novel methodology, the cohort differences model, to predict whether a patient belongs to the training or to the validation cohort. If the cohort differences model performs well, it would suggest a large difference in cohort characteristics meaning we would validate the transferability of the model rather than reproducibility. We tested our method in a prospective validation of three existing models for pCR prediction in 154 patients. Results: Our results showed a large difference between training and validation cohort for one of the three tested models [Area under the Receiver Operating Curve (AUC) cohort differences model: 0.85], signaling the validation leans towards transferability. Two out of three models had a lower AUC for validation (0.66 and 0.58), one model showed a higher AUC in the validation cohort (0.70). Discussion: We have successfully applied a new methodology in the validation of three prediction models, which allows us to indicate if a validation targeted transferability (large differences between training/validation cohort) or reproducibility (small cohort differences).",
author = "{van Soest}, Johan and Elisa Meldolesi and {van Stiphout}, Ruud and Roberto Gatta and Andrea Damiani and Vincenzo Valentini and Philippe Lambin and Andre Dekker",
year = "2017",
doi = "10.1002/mp.12423",
language = "English",
volume = "44",
journal = "Medical Physics",
issn = "0094-2405",
publisher = "Wiley",
number = "9",

}

Prospective validation of pathologic complete response models in rectal cancer : Transferability and reproducibility. / van Soest, Johan; Meldolesi, Elisa; van Stiphout, Ruud; Gatta, Roberto; Damiani, Andrea; Valentini, Vincenzo; Lambin, Philippe; Dekker, Andre.

In: Medical Physics, Vol. 44, No. 9, 2017.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Prospective validation of pathologic complete response models in rectal cancer

T2 - Transferability and reproducibility

AU - van Soest, Johan

AU - Meldolesi, Elisa

AU - van Stiphout, Ruud

AU - Gatta, Roberto

AU - Damiani, Andrea

AU - Valentini, Vincenzo

AU - Lambin, Philippe

AU - Dekker, Andre

PY - 2017

Y1 - 2017

N2 - Purpose: Multiple models have been developed to predict pathologic complete response (pCR) in locally advanced rectal cancer patients. Unfortunately, validation of these models normally omit the implications of cohort differences on prediction model performance. In this work, we will perform a prospective validation of three pCR models, including information whether this validation will target transferability or reproducibility (cohort differences) of the given models. Methods: We applied a novel methodology, the cohort differences model, to predict whether a patient belongs to the training or to the validation cohort. If the cohort differences model performs well, it would suggest a large difference in cohort characteristics meaning we would validate the transferability of the model rather than reproducibility. We tested our method in a prospective validation of three existing models for pCR prediction in 154 patients. Results: Our results showed a large difference between training and validation cohort for one of the three tested models [Area under the Receiver Operating Curve (AUC) cohort differences model: 0.85], signaling the validation leans towards transferability. Two out of three models had a lower AUC for validation (0.66 and 0.58), one model showed a higher AUC in the validation cohort (0.70). Discussion: We have successfully applied a new methodology in the validation of three prediction models, which allows us to indicate if a validation targeted transferability (large differences between training/validation cohort) or reproducibility (small cohort differences).

AB - Purpose: Multiple models have been developed to predict pathologic complete response (pCR) in locally advanced rectal cancer patients. Unfortunately, validation of these models normally omit the implications of cohort differences on prediction model performance. In this work, we will perform a prospective validation of three pCR models, including information whether this validation will target transferability or reproducibility (cohort differences) of the given models. Methods: We applied a novel methodology, the cohort differences model, to predict whether a patient belongs to the training or to the validation cohort. If the cohort differences model performs well, it would suggest a large difference in cohort characteristics meaning we would validate the transferability of the model rather than reproducibility. We tested our method in a prospective validation of three existing models for pCR prediction in 154 patients. Results: Our results showed a large difference between training and validation cohort for one of the three tested models [Area under the Receiver Operating Curve (AUC) cohort differences model: 0.85], signaling the validation leans towards transferability. Two out of three models had a lower AUC for validation (0.66 and 0.58), one model showed a higher AUC in the validation cohort (0.70). Discussion: We have successfully applied a new methodology in the validation of three prediction models, which allows us to indicate if a validation targeted transferability (large differences between training/validation cohort) or reproducibility (small cohort differences).

U2 - 10.1002/mp.12423

DO - 10.1002/mp.12423

M3 - Article

VL - 44

JO - Medical Physics

JF - Medical Physics

SN - 0094-2405

IS - 9

ER -