Reliability of narrative assessment data on communication skills in a summative OSCE

Kyle John Wilby*, Marjan J. B. Govaerts, Diana H. J. M. Dolmans, Zubin Austin, Cees van der Vleuten

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review

4 Citations (Web of Science)
33 Downloads (Pure)

Abstract

Objective: To quantitatively estimate the reliability of narrative assessment data regarding student communication skills obtained from a summative OSCE and to compare reliability to that of communication scores obtained from direct observation.

Methods: Narrative comments and communication scores (scale 1-5) were obtained for 14 graduating pharmacy students across 6 summative OSCE stations with 2 assessors per station who directly observed student performance. Two assessors who had not observed the OSCE reviewed narratives and independently scored communication skills according to the same 5-point scale. Generalizability theory was used to estimate reliability. Correlation was used to evaluate the relationship between scores from each assessment method.

Results: A total of 168 narratives and communication scores were obtained. The G-coefficients were 0.571 for scores provided by assessors present during the OSCE and 0.612 for scores from assessors who provided scores based on narratives only. Correlation between the two sets of scores was 0.5.

Conclusion: Reliability of communication scores is not dependent on whether assessors directly observe student performance or assess written narratives, yet both conditions appear to measure communication skills somewhat differently.

Practice implications: Narratives may be useful for summative decision-making and help overcome the current limitations of using solely quantitative scores. (C) 2019 Elsevier B.V. All rights reserved.

Original languageEnglish
Pages (from-to)1164-1169
Number of pages6
JournalPatient Education and Counseling
Volume102
Issue number6
DOIs
Publication statusPublished - Jun 2019

Keywords

  • Assessment
  • Communication
  • Medical education
  • Pharmacy education
  • STRUCTURED CLINICAL EXAMINATION
  • HEALTH OUTCOMES
  • PERFORMANCE
  • STUDENTS
  • REPRODUCIBILITY
  • CHECKLISTS
  • COMPETENCE

Cite this