Effect of CAD on performance in ASPECTS reading

Marielle Ernst; Martina Bernhardt; Matthias Bechstein; Gerhard Schön; Jens Fiehler; Charles B.L.M. Majoie; Henk A. Marquering; Wim H. van Zwam; Diederik W.J. Dippel; Robert J. van Oostenbrugge; Einar Goebell

doi:10.1016/j.imu.2020.100295

Effect of CAD on performance in ASPECTS reading

Marielle Ernst^*, Martina Bernhardt, Matthias Bechstein, Gerhard Schön, Jens Fiehler, Charles B.L.M. Majoie, Henk A. Marquering, Wim H. van Zwam, Diederik W.J. Dippel, Robert J. van Oostenbrugge, Einar Goebell

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

Background and purpose: While computer-aided diagnosis (CAD) tools are widely used in stroke imaging routines already, their influence on actual decision-making is still underexplored. We analyzed the effect of a simulated CAD tool on ASPECT-Scoring on acute-stroke CT-scans with respect to experience level. Materials and methods: Baseline CT scans of 100 stroke patients from the MR CLEAN trial with consensus-ASPECTS as ground truth were independently ASPECTS graded by three readers with different levels of experience. Weeks later the same CTs were re-analyzed with additional displaying of simulated ASPECTS (s-ASPECTS, by adding or subtracting 2 points from the ground truth). Readers were told that the score was generated by an automatic ASPECT-Scoring algorithm. The influence of the displayed s-ASPECTS on the readers’ second ASPECT-Scoring was analyzed by using a linear mixed model and the reliability was assessed. Performance was measured as the absolute difference between readers ASPECTS and consensus-ASPECTS. Results: The influence of the s-ASPECTS on the second ASPECT-Scoring was the lowest for the reader with the most experience in neuroradiology, while the other readers were significantly more influenced. All readers veered further away from the ground truth in their second ASPECT-Scoring with the s-ASPECTS, though not significantly. Overall interrater reliability was excellent (ICC = 0.94 [0.92–0.96]). Conclusions: ASPECT-Scoring may be significantly influenced by simulated ASPECTS displayed by a suboptimal CAD tool, especially in readers with less experience, and performance tends to decrease.

Original language	English
Article number	100295
Number of pages	8
Journal	Informatics in Medicine Unlocked
Volume	18
DOIs	https://doi.org/10.1016/j.imu.2020.100295
Publication status	Published - 1 Jan 2020

Keywords

Alberta stroke program early CT score
Computed tomography
Diagnostic method
Ischemic stroke
Machine learning

Access to Document

10.1016/j.imu.2020.100295Licence: CC BY-NC-ND

Cite this

@article{9176cb8295cb4dfe8f3e31811f2866e9,

title = "Effect of CAD on performance in ASPECTS reading",

abstract = "Background and purpose: While computer-aided diagnosis (CAD) tools are widely used in stroke imaging routines already, their influence on actual decision-making is still underexplored. We analyzed the effect of a simulated CAD tool on ASPECT-Scoring on acute-stroke CT-scans with respect to experience level. Materials and methods: Baseline CT scans of 100 stroke patients from the MR CLEAN trial with consensus-ASPECTS as ground truth were independently ASPECTS graded by three readers with different levels of experience. Weeks later the same CTs were re-analyzed with additional displaying of simulated ASPECTS (s-ASPECTS, by adding or subtracting 2 points from the ground truth). Readers were told that the score was generated by an automatic ASPECT-Scoring algorithm. The influence of the displayed s-ASPECTS on the readers{\textquoteright} second ASPECT-Scoring was analyzed by using a linear mixed model and the reliability was assessed. Performance was measured as the absolute difference between readers ASPECTS and consensus-ASPECTS. Results: The influence of the s-ASPECTS on the second ASPECT-Scoring was the lowest for the reader with the most experience in neuroradiology, while the other readers were significantly more influenced. All readers veered further away from the ground truth in their second ASPECT-Scoring with the s-ASPECTS, though not significantly. Overall interrater reliability was excellent (ICC = 0.94 [0.92–0.96]). Conclusions: ASPECT-Scoring may be significantly influenced by simulated ASPECTS displayed by a suboptimal CAD tool, especially in readers with less experience, and performance tends to decrease.",

keywords = "Alberta stroke program early CT score, Computed tomography, Diagnostic method, Ischemic stroke, Machine learning",

author = "Marielle Ernst and Martina Bernhardt and Matthias Bechstein and Gerhard Sch{\"o}n and Jens Fiehler and Majoie, {Charles B.L.M.} and Marquering, {Henk A.} and {van Zwam}, {Wim H.} and Dippel, {Diederik W.J.} and {van Oostenbrugge}, {Robert J.} and Einar Goebell",

note = "Funding Information: DWJD reports grants from the Dutch Heart Foundation , Medtronic/Covidien , AngioCare , Stryker , Medac , Lamepro , Penumbra , Top Medical/Concentric . WHvZ reports speaking engagements with Stryker and Codman. HAM reports ownership interest with Nico-lab. CBM reports research grants from the Dutch Heart Foundation for the submitted work; Toegepast Wetenschappelijk Instituut voor Neuromodulatie foundation , European Commission and Stryker (outside the submitted work); all paid to institution. The other authors declare no competing interests. Publisher Copyright: {\textcopyright} 2020 The Authors",

year = "2020",

month = jan,

day = "1",

doi = "10.1016/j.imu.2020.100295",

language = "English",

volume = "18",

journal = "Informatics in Medicine Unlocked",

issn = "2352-9148",

publisher = "Elsevier Ltd",

}

TY - JOUR

T1 - Effect of CAD on performance in ASPECTS reading

AU - Ernst, Marielle

AU - Bernhardt, Martina

AU - Bechstein, Matthias

AU - Schön, Gerhard

AU - Fiehler, Jens

AU - Majoie, Charles B.L.M.

AU - Marquering, Henk A.

AU - van Zwam, Wim H.

AU - Dippel, Diederik W.J.

AU - van Oostenbrugge, Robert J.

AU - Goebell, Einar

N1 - Funding Information: DWJD reports grants from the Dutch Heart Foundation , Medtronic/Covidien , AngioCare , Stryker , Medac , Lamepro , Penumbra , Top Medical/Concentric . WHvZ reports speaking engagements with Stryker and Codman. HAM reports ownership interest with Nico-lab. CBM reports research grants from the Dutch Heart Foundation for the submitted work; Toegepast Wetenschappelijk Instituut voor Neuromodulatie foundation , European Commission and Stryker (outside the submitted work); all paid to institution. The other authors declare no competing interests. Publisher Copyright: © 2020 The Authors

PY - 2020/1/1

Y1 - 2020/1/1

N2 - Background and purpose: While computer-aided diagnosis (CAD) tools are widely used in stroke imaging routines already, their influence on actual decision-making is still underexplored. We analyzed the effect of a simulated CAD tool on ASPECT-Scoring on acute-stroke CT-scans with respect to experience level. Materials and methods: Baseline CT scans of 100 stroke patients from the MR CLEAN trial with consensus-ASPECTS as ground truth were independently ASPECTS graded by three readers with different levels of experience. Weeks later the same CTs were re-analyzed with additional displaying of simulated ASPECTS (s-ASPECTS, by adding or subtracting 2 points from the ground truth). Readers were told that the score was generated by an automatic ASPECT-Scoring algorithm. The influence of the displayed s-ASPECTS on the readers’ second ASPECT-Scoring was analyzed by using a linear mixed model and the reliability was assessed. Performance was measured as the absolute difference between readers ASPECTS and consensus-ASPECTS. Results: The influence of the s-ASPECTS on the second ASPECT-Scoring was the lowest for the reader with the most experience in neuroradiology, while the other readers were significantly more influenced. All readers veered further away from the ground truth in their second ASPECT-Scoring with the s-ASPECTS, though not significantly. Overall interrater reliability was excellent (ICC = 0.94 [0.92–0.96]). Conclusions: ASPECT-Scoring may be significantly influenced by simulated ASPECTS displayed by a suboptimal CAD tool, especially in readers with less experience, and performance tends to decrease.

AB - Background and purpose: While computer-aided diagnosis (CAD) tools are widely used in stroke imaging routines already, their influence on actual decision-making is still underexplored. We analyzed the effect of a simulated CAD tool on ASPECT-Scoring on acute-stroke CT-scans with respect to experience level. Materials and methods: Baseline CT scans of 100 stroke patients from the MR CLEAN trial with consensus-ASPECTS as ground truth were independently ASPECTS graded by three readers with different levels of experience. Weeks later the same CTs were re-analyzed with additional displaying of simulated ASPECTS (s-ASPECTS, by adding or subtracting 2 points from the ground truth). Readers were told that the score was generated by an automatic ASPECT-Scoring algorithm. The influence of the displayed s-ASPECTS on the readers’ second ASPECT-Scoring was analyzed by using a linear mixed model and the reliability was assessed. Performance was measured as the absolute difference between readers ASPECTS and consensus-ASPECTS. Results: The influence of the s-ASPECTS on the second ASPECT-Scoring was the lowest for the reader with the most experience in neuroradiology, while the other readers were significantly more influenced. All readers veered further away from the ground truth in their second ASPECT-Scoring with the s-ASPECTS, though not significantly. Overall interrater reliability was excellent (ICC = 0.94 [0.92–0.96]). Conclusions: ASPECT-Scoring may be significantly influenced by simulated ASPECTS displayed by a suboptimal CAD tool, especially in readers with less experience, and performance tends to decrease.

KW - Alberta stroke program early CT score

KW - Computed tomography

KW - Diagnostic method

KW - Ischemic stroke

KW - Machine learning

U2 - 10.1016/j.imu.2020.100295

DO - 10.1016/j.imu.2020.100295

M3 - Article

SN - 2352-9148

VL - 18

JO - Informatics in Medicine Unlocked

JF - Informatics in Medicine Unlocked

M1 - 100295

ER -

Effect of CAD on performance in ASPECTS reading

Abstract

Keywords

Access to Document

Fingerprint

Cite this