External Validation of a Bayesian Network for Error Detection in Radiotherapy Plans

Artificial intelligence (AI) applications have recently been proposed to detect errors in radiotherapy plans. External validation of such systems is essential to assess their performance and safety before applying them to clinical practice. We collected data from 5238 patients treated at Maastro Clinic and introduced a range of common radiotherapy plan errors for the model to detect. We estimated the model's discrimination by calculating the area under the receiver-operating characteristic curve (AUC). We also assessed its clinical usefulness as an alert system that could reduce the need for manual checks by calculating the percentage of values flagged as errors and the positive predictive value (PPV) for a range of high sensitivities (95 %-99 %) and error prevalence. The AUC when considering all variables was 67.8% (95% CI, 65.6%-69.9%). The AUC varied widely for different types of errors (from 90.4% for table angle errors to 54.5% for planning tumor volume-PTV dose errors). The percentage of flagged values ranged from 84% to 90% for sensitivities between 95% and 99% and the PPV was only slightly higher than the prevalence of the errors. The model's performance in the external validation was significantly worse than that in its original setting (AUC of 68% versus 89%). Its usefulness as an alert system to reduce the need for manual checks is questionable due to the low PPV and high percentage of values flagged as potential errors to achieve a high sensitivity. We analyzed the apparent limitations of the model and we proposed actions to overcome them.