Validity evidence for summative performance evaluations in postgraduate community pharmacy education

Marnix P D Westein; Andries S Koster; Hester E M Daelmans; Carlos F Collares; Marcel L Bouvy; Rashmi A Kusurkar

doi:10.1016/j.cptl.2022.06.014

Validity evidence for summative performance evaluations in postgraduate community pharmacy education

Marnix P D Westein^*, Andries S Koster, Hester E M Daelmans, Carlos F Collares, Marcel L Bouvy, Rashmi A Kusurkar

^*Corresponding author for this work

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

INTRODUCTION: Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework.

METHODS: The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process.

RESULTS: A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director.

CONCLUSIONS: The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.

Original language	English
Pages (from-to)	701-711
Number of pages	11
Journal	Currents in Pharmacy Teaching and Learning
Volume	14
Issue number	6
DOIs	https://doi.org/10.1016/j.cptl.2022.06.014
Publication status	Published - Jun 2022

Keywords

Canada
Clinical Competence
Educational Measurement
Humans
Pharmacies
Reproducibility of Results
Supervisor
Performance evaluation
Decision making
PROGRAMMATIC ASSESSMENT
COMPETENCES
MEDICAL-EDUCATION
GUIDE
CanMEDS
CARE
CHALLENGES
FIT
Workplace -based assessment
Shadow assessment system
GENERALIZABILITY

Access to Document

10.1016/j.cptl.2022.06.014Licence: CC BY

Cite this

@article{184ff86dab7441c4be4b19fe2d20f132,

title = "Validity evidence for summative performance evaluations in postgraduate community pharmacy education",

abstract = "INTRODUCTION: Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework.METHODS: The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process.RESULTS: A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director.CONCLUSIONS: The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.",

keywords = "Canada, Clinical Competence, Educational Measurement, Humans, Pharmacies, Reproducibility of Results, Supervisor, Performance evaluation, Decision making, PROGRAMMATIC ASSESSMENT, COMPETENCES, MEDICAL-EDUCATION, GUIDE, CanMEDS, CARE, CHALLENGES, FIT, Workplace -based assessment, Shadow assessment system, GENERALIZABILITY",

author = "Westein, {Marnix P D} and Koster, {Andries S} and Daelmans, {Hester E M} and Collares, {Carlos F} and Bouvy, {Marcel L} and Kusurkar, {Rashmi A}",

year = "2022",

month = jun,

doi = "10.1016/j.cptl.2022.06.014",

language = "English",

volume = "14",

pages = "701--711",

journal = "Currents in Pharmacy Teaching and Learning",

issn = "1877-1297",

publisher = "Elsevier Science",

number = "6",

}

TY - JOUR

T1 - Validity evidence for summative performance evaluations in postgraduate community pharmacy education

AU - Westein, Marnix P D

AU - Koster, Andries S

AU - Daelmans, Hester E M

AU - Collares, Carlos F

AU - Bouvy, Marcel L

AU - Kusurkar, Rashmi A

PY - 2022/6

Y1 - 2022/6

N2 - INTRODUCTION: Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework.METHODS: The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process.RESULTS: A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director.CONCLUSIONS: The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.

AB - INTRODUCTION: Workplace-based assessment of competencies is complex. In this study, the validity of summative performance evaluations (SPEs) made by supervisors in a two-year longitudinal supervisor-trainee relationship was investigated in a postgraduate community pharmacy specialization program in the Netherlands. The construct of competence was based on an adapted version of the 2005 Canadian Medical Education Directive for Specialists (CanMEDS) framework.METHODS: The study had a case study design. Both quantitative and qualitative data were collected. The year 1 and year 2 SPE scores of 342 trainees were analyzed using confirmatory factor analysis and generalizability theory. Semi-structured interviews were held with 15 supervisors and the program director to analyze the inferences they made and the impact of SPE scores on the decision-making process.RESULTS: A good model fit was found for the adapted CanMEDS based seven-factor construct. The reliability/precision of the SPE measurements could not be completely isolated, as every trainee was trained in one pharmacy and evaluated by one supervisor. Qualitative analysis revealed that supervisors varied in their standards for scoring competencies. Some supervisors were reluctant to fail trainees. The competency scores had little impact on the high-stakes decision made by the program director.CONCLUSIONS: The adapted CanMEDS competency framework provided a valid structure to measure competence. The reliability/precision of SPE measurements could not be established and the SPE measurements provided limited input for the decision-making process. Indications of a shadow assessment system in the pharmacies need further investigation.

KW - Canada

KW - Clinical Competence

KW - Educational Measurement

KW - Humans

KW - Pharmacies

KW - Reproducibility of Results

KW - Supervisor

KW - Performance evaluation

KW - Decision making

KW - PROGRAMMATIC ASSESSMENT

KW - COMPETENCES

KW - MEDICAL-EDUCATION

KW - GUIDE

KW - CanMEDS

KW - CARE

KW - CHALLENGES

KW - FIT

KW - Workplace -based assessment

KW - Shadow assessment system

KW - GENERALIZABILITY

U2 - 10.1016/j.cptl.2022.06.014

DO - 10.1016/j.cptl.2022.06.014

M3 - Article

C2 - 35809899

SN - 1877-1297

VL - 14

SP - 701

EP - 711

JO - Currents in Pharmacy Teaching and Learning

JF - Currents in Pharmacy Teaching and Learning

IS - 6

ER -