STAPLE performance assessed on crowdsourced sclera segmentations

Malte-Levin Jauer; Saksham Goel; Yash Sharma; Thomas M. Deserno; Marlies Gijs; Tos T. J. M. Berendschot; Christian J. F. Bertens; Rudy M. M. A. Nuijts

doi:10.1117/12.2551297

STAPLE performance assessed on crowdsourced sclera segmentations

Malte-Levin Jauer^*, Saksham Goel, Yash Sharma, Thomas M. Deserno, Marlies Gijs, Tos T. J. M. Berendschot, Christian J. F. Bertens, Rudy M. M. A. Nuijts

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

1 Citation (Web of Science)

43 Downloads (Pure)

Abstract

The Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm is frequently used in medical image segmentation without available ground truth (GT). In this paper, we investigate the number of inexperienced users required to establish a reliable STAPLE-based GT and the number of vertices the user's shall place for a point-based segmentation. We employ "WeLineation", a novel web-based system for crowdsourcing segmentations. Within the study, 2,060 masks have been delivered by 44 users on 75 different photographic images of the human eye, where users had to segment the sclera. For all masks, GT was estimated using STAPLE. Then, STAPLE is computed using fewer user contributions and results are compared to the GT. Requiring an error rate lower than 2%, same segmentation performance is obtained with 13 experienced and 22 rather inexperienced users. More than 10 vertices shall be placed on the delineation contour in order to reach an accuracy larger than 95%. In average, a vertex along the segmentation contour shall be placed every 81 pixels. The results indicate that knowledge about the users performance can reduce the number of segmentation masks per image, which are needed to estimate reliable GT. Therefore, gathering performance parameters of users during a crowdsourcing study and applying this information to the assignment process is recommended. In this way, benefits in the cost-effectiveness of a crowdsourcing segmentation study can be achieved.

Original language	English
Title of host publication	MEDICAL IMAGING 2020: IMAGING INFORMATICS FOR HEALTHCARE, RESEARCH, AND APPLICATIONS
Editors	PH Chen, TM Deserno
Publisher	SPIE-INT SOC OPTICAL ENGINEERING
Number of pages	6
Volume	11318
ISBN (Print)	978-1-5106-3403-9
DOIs	https://doi.org/10.1117/12.2551297
Publication status	Published - 2020
Event	SPIE Conference on Medical Imaging - Imaging Informatics for Healthcare, Research, and Applications - Houston, United States Duration: 16 Feb 2020 → 17 Feb 2020

Publication series

Series	Proceedings of SPIE
Volume	11318
ISSN	0277-786X

Conference

Conference	SPIE Conference on Medical Imaging - Imaging Informatics for Healthcare, Research, and Applications
Country/Territory	United States
City	Houston
Period	16/02/20 → 17/02/20

Keywords

Segmentation
Crowdsourcing
Ground Truth
STAPLE
Image Processing
Sclera
SIMULTANEOUS TRUTH
VALIDATION
ALGORITHM

Access to Document

10.1117/12.2551297

Full TextFinal published version, 4.02 MBLicence: Taverne

Cite this

Jauer, M-L., Goel, S., Sharma, Y., Deserno, T. M., Gijs, M., Berendschot, T. T. J. M., Bertens, C. J. F., & Nuijts, R. M. M. A. (2020). STAPLE performance assessed on crowdsourced sclera segmentations. In PH. Chen, & TM. Deserno (Eds.), MEDICAL IMAGING 2020: IMAGING INFORMATICS FOR HEALTHCARE, RESEARCH, AND APPLICATIONS (Vol. 11318). [113180K] SPIE-INT SOC OPTICAL ENGINEERING. Proceedings of SPIE Vol. 11318 https://doi.org/10.1117/12.2551297

@inproceedings{d51b3989e1354239897452321e7ce972,

title = "STAPLE performance assessed on crowdsourced sclera segmentations",

abstract = "The Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm is frequently used in medical image segmentation without available ground truth (GT). In this paper, we investigate the number of inexperienced users required to establish a reliable STAPLE-based GT and the number of vertices the user's shall place for a point-based segmentation. We employ {"}WeLineation{"}, a novel web-based system for crowdsourcing segmentations. Within the study, 2,060 masks have been delivered by 44 users on 75 different photographic images of the human eye, where users had to segment the sclera. For all masks, GT was estimated using STAPLE. Then, STAPLE is computed using fewer user contributions and results are compared to the GT. Requiring an error rate lower than 2%, same segmentation performance is obtained with 13 experienced and 22 rather inexperienced users. More than 10 vertices shall be placed on the delineation contour in order to reach an accuracy larger than 95%. In average, a vertex along the segmentation contour shall be placed every 81 pixels. The results indicate that knowledge about the users performance can reduce the number of segmentation masks per image, which are needed to estimate reliable GT. Therefore, gathering performance parameters of users during a crowdsourcing study and applying this information to the assignment process is recommended. In this way, benefits in the cost-effectiveness of a crowdsourcing segmentation study can be achieved.",

keywords = "Segmentation, Crowdsourcing, Ground Truth, STAPLE, Image Processing, Sclera, SIMULTANEOUS TRUTH, VALIDATION, ALGORITHM",

author = "Malte-Levin Jauer and Saksham Goel and Yash Sharma and Deserno, {Thomas M.} and Marlies Gijs and Berendschot, {Tos T. J. M.} and Bertens, {Christian J. F.} and Nuijts, {Rudy M. M. A.}",

year = "2020",

doi = "10.1117/12.2551297",

language = "English",

isbn = "978-1-5106-3403-9",

volume = "11318",

series = "Proceedings of SPIE",

publisher = "SPIE-INT SOC OPTICAL ENGINEERING",

editor = "PH Chen and TM Deserno",

booktitle = "MEDICAL IMAGING 2020: IMAGING INFORMATICS FOR HEALTHCARE, RESEARCH, AND APPLICATIONS",

note = "SPIE Conference on Medical Imaging - Imaging Informatics for Healthcare, Research, and Applications ; Conference date: 16-02-2020 Through 17-02-2020",

}

Jauer, M-L, Goel, S, Sharma, Y, Deserno, TM, Gijs, M , Berendschot, TTJM, Bertens, CJF & Nuijts, RMMA 2020, STAPLE performance assessed on crowdsourced sclera segmentations. in PH Chen & TM Deserno (eds), MEDICAL IMAGING 2020: IMAGING INFORMATICS FOR HEALTHCARE, RESEARCH, AND APPLICATIONS. vol. 11318, 113180K, SPIE-INT SOC OPTICAL ENGINEERING, Proceedings of SPIE, vol. 11318, SPIE Conference on Medical Imaging - Imaging Informatics for Healthcare, Research, and Applications, Houston, Texas, United States, 16/02/20. https://doi.org/10.1117/12.2551297

STAPLE performance assessed on crowdsourced sclera segmentations. / Jauer, Malte-Levin; Goel, Saksham; Sharma, Yash et al.
MEDICAL IMAGING 2020: IMAGING INFORMATICS FOR HEALTHCARE, RESEARCH, AND APPLICATIONS. ed. / PH Chen; TM Deserno. Vol. 11318 SPIE-INT SOC OPTICAL ENGINEERING, 2020. 113180K (Proceedings of SPIE, Vol. 11318).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - STAPLE performance assessed on crowdsourced sclera segmentations

AU - Jauer, Malte-Levin

AU - Goel, Saksham

AU - Sharma, Yash

AU - Deserno, Thomas M.

AU - Gijs, Marlies

AU - Berendschot, Tos T. J. M.

AU - Bertens, Christian J. F.

AU - Nuijts, Rudy M. M. A.

PY - 2020

Y1 - 2020

N2 - The Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm is frequently used in medical image segmentation without available ground truth (GT). In this paper, we investigate the number of inexperienced users required to establish a reliable STAPLE-based GT and the number of vertices the user's shall place for a point-based segmentation. We employ "WeLineation", a novel web-based system for crowdsourcing segmentations. Within the study, 2,060 masks have been delivered by 44 users on 75 different photographic images of the human eye, where users had to segment the sclera. For all masks, GT was estimated using STAPLE. Then, STAPLE is computed using fewer user contributions and results are compared to the GT. Requiring an error rate lower than 2%, same segmentation performance is obtained with 13 experienced and 22 rather inexperienced users. More than 10 vertices shall be placed on the delineation contour in order to reach an accuracy larger than 95%. In average, a vertex along the segmentation contour shall be placed every 81 pixels. The results indicate that knowledge about the users performance can reduce the number of segmentation masks per image, which are needed to estimate reliable GT. Therefore, gathering performance parameters of users during a crowdsourcing study and applying this information to the assignment process is recommended. In this way, benefits in the cost-effectiveness of a crowdsourcing segmentation study can be achieved.

AB - The Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm is frequently used in medical image segmentation without available ground truth (GT). In this paper, we investigate the number of inexperienced users required to establish a reliable STAPLE-based GT and the number of vertices the user's shall place for a point-based segmentation. We employ "WeLineation", a novel web-based system for crowdsourcing segmentations. Within the study, 2,060 masks have been delivered by 44 users on 75 different photographic images of the human eye, where users had to segment the sclera. For all masks, GT was estimated using STAPLE. Then, STAPLE is computed using fewer user contributions and results are compared to the GT. Requiring an error rate lower than 2%, same segmentation performance is obtained with 13 experienced and 22 rather inexperienced users. More than 10 vertices shall be placed on the delineation contour in order to reach an accuracy larger than 95%. In average, a vertex along the segmentation contour shall be placed every 81 pixels. The results indicate that knowledge about the users performance can reduce the number of segmentation masks per image, which are needed to estimate reliable GT. Therefore, gathering performance parameters of users during a crowdsourcing study and applying this information to the assignment process is recommended. In this way, benefits in the cost-effectiveness of a crowdsourcing segmentation study can be achieved.

KW - Segmentation

KW - Crowdsourcing

KW - Ground Truth

KW - STAPLE

KW - Image Processing

KW - Sclera

KW - SIMULTANEOUS TRUTH

KW - VALIDATION

KW - ALGORITHM

U2 - 10.1117/12.2551297

DO - 10.1117/12.2551297

M3 - Conference article in proceeding

SN - 978-1-5106-3403-9

VL - 11318

T3 - Proceedings of SPIE

BT - MEDICAL IMAGING 2020: IMAGING INFORMATICS FOR HEALTHCARE, RESEARCH, AND APPLICATIONS

A2 - Chen, PH

A2 - Deserno, TM

PB - SPIE-INT SOC OPTICAL ENGINEERING

T2 - SPIE Conference on Medical Imaging - Imaging Informatics for Healthcare, Research, and Applications

Y2 - 16 February 2020 through 17 February 2020

ER -

Jauer M-L, Goel S, Sharma Y, Deserno TM, Gijs M , Berendschot TTJM et al. STAPLE performance assessed on crowdsourced sclera segmentations. In Chen PH, Deserno TM, editors, MEDICAL IMAGING 2020: IMAGING INFORMATICS FOR HEALTHCARE, RESEARCH, AND APPLICATIONS. Vol. 11318. SPIE-INT SOC OPTICAL ENGINEERING. 2020. 113180K. (Proceedings of SPIE, Vol. 11318). doi: 10.1117/12.2551297