Questioning the rater idiosyncrasy explanation for error variance by searching for multiple signals within the noise

A.M. Gingerich

doi:10.26481/dis.20150903ag

Questioning the rater idiosyncrasy explanation for error variance by searching for multiple signals within the noise

A.M. Gingerich

FHML non-thematic output

Research output: Thesis › Doctoral Thesis › External prepared

623 Downloads (Pure)

Abstract

Medical trainees are assessed performing clinical tasks but the examiners’ ratings can be highly variable. It is assumed that examiners assessing the same performance should form similar judgments and provide similar ratings. As such, the psychometric models currently used to analyze the ratings assume there is a single point of consensus. This research, however, found multiple clusters of consensus within the variable assessments provided by examiners for a single performance. This finding was consistent across two samples of participants and two different methodologies. Finding more than one point of consensus challenges the use of psychometric models to analyze examiners’ ratings.

Original language	English
Qualification	Doctor of Philosophy
Awarding Institution	Maastricht University
Supervisors/Advisors	van der Vleuten, Cornelis, Supervisor Eva, K.W., Supervisor, External person Regehr, Glenn, Supervisor, External person
Award date	3 Sept 2015
Place of Publication	Maastricht
Publisher	Datawyse / Universitaire Pers Maastricht
Print ISBNs	9789461594556
DOIs	https://doi.org/10.26481/dis.20150903ag
Publication status	Published - 2015

Keywords

rater cognition
medical education
rater-based assessment

Access to Document

10.26481/dis.20150903ag

Full TextFinal published version, 1.21 MB
AbstractFinal published version, 87.7 KB
PropositionsFinal published version, 25.2 KB
CoverFinal published version, 92.9 KB
ValorisationFinal published version, 30.3 KB

Cite this

@phdthesis{ebfb907fa63c4b75b3b51fff3694150d,

title = "Questioning the rater idiosyncrasy explanation for error variance by searching for multiple signals within the noise",

abstract = "Medical trainees are assessed performing clinical tasks but the examiners{\textquoteright} ratings can be highly variable. It is assumed that examiners assessing the same performance should form similar judgments and provide similar ratings. As such, the psychometric models currently used to analyze the ratings assume there is a single point of consensus. This research, however, found multiple clusters of consensus within the variable assessments provided by examiners for a single performance. This finding was consistent across two samples of participants and two different methodologies. Finding more than one point of consensus challenges the use of psychometric models to analyze examiners{\textquoteright} ratings. ",

keywords = "rater cognition, medical education, rater-based assessment",

author = "A.M. Gingerich",

year = "2015",

doi = "10.26481/dis.20150903ag",

language = "English",

isbn = "9789461594556",

publisher = "Datawyse / Universitaire Pers Maastricht",

address = "Netherlands",

school = "Maastricht University",

}

TY - BOOK

T1 - Questioning the rater idiosyncrasy explanation for error variance by searching for multiple signals within the noise

AU - Gingerich, A.M.

PY - 2015

Y1 - 2015

N2 - Medical trainees are assessed performing clinical tasks but the examiners’ ratings can be highly variable. It is assumed that examiners assessing the same performance should form similar judgments and provide similar ratings. As such, the psychometric models currently used to analyze the ratings assume there is a single point of consensus. This research, however, found multiple clusters of consensus within the variable assessments provided by examiners for a single performance. This finding was consistent across two samples of participants and two different methodologies. Finding more than one point of consensus challenges the use of psychometric models to analyze examiners’ ratings.

AB - Medical trainees are assessed performing clinical tasks but the examiners’ ratings can be highly variable. It is assumed that examiners assessing the same performance should form similar judgments and provide similar ratings. As such, the psychometric models currently used to analyze the ratings assume there is a single point of consensus. This research, however, found multiple clusters of consensus within the variable assessments provided by examiners for a single performance. This finding was consistent across two samples of participants and two different methodologies. Finding more than one point of consensus challenges the use of psychometric models to analyze examiners’ ratings.

KW - rater cognition

KW - medical education

KW - rater-based assessment

U2 - 10.26481/dis.20150903ag

DO - 10.26481/dis.20150903ag

M3 - Doctoral Thesis

SN - 9789461594556

PB - Datawyse / Universitaire Pers Maastricht

CY - Maastricht

ER -