TY - JOUR
T1 - The development and implementation of a computer adaptive progress test across European countries
AU - Rice, Neil
AU - Pêgo, José Miguel
AU - Collares, Carlos
AU - Kisielewska, Jolanta
AU - Gale, Thomas
N1 - Funding Information:
The authors wish to thank Annemarie Camp, who project managed the consortium development and pilot test administrations, and all members of the OAIPT consortium who have contributed to the outputs of the project to date: Jagiellonian University, Poland (Michal Nowakowski, Mateusz Rubinkiewicz); Maastricht University, The Netherlands (Evelien Neis, Joyce Moonen-van Loon, Cees van der Vleuten); Medical University of Lodz, Poland (Janusz Janczukowicz, Katarzyna Janusz, Paulina Sobieranska); University of Exeter, UK (Kevin Brandom, Adrian Freeman, Jonathan Wyatt); University of Helsinki, Finland (Otto Helve, Mika Laitenan, Johanna Louhimo); University Medical Center Groningen, The Netherlands (Debbie Jaarsma, Bram Jacobs, Michiel Katoele, Ally van Hell); University of Minho, Portugal (Nuno Santos); and University of Plymouth, UK (Steven Burr, Ian Hodgins). This study was supported by the Erasmus+ grant 2018-1-NLOI-KA203-038925.
Publisher Copyright:
© 2022 The Authors
PY - 2022/6/6
Y1 - 2022/6/6
N2 - Longitudinal progress testing promotes self-directed deep learning across a full spectrum of knowledge, enabling early detection of underperformance and opportunities for remediation. Computer adaptive testing (CAT), where the difficulty of a test dynamically adjusts according to a test taker's ability, has benefits in a progress testing context, but significant resource and experience is required to develop appropriate test materials. This study describes how a transnational consortium from eight medical schools in five countries across Europe was formed to develop a computer adaptive progress test applicable across international curricula. 1,212 students from more than 40 nationalities took part in the study, of whom more than 70% were not native English speakers, though nearly all reported competence in English. A content map for an international assessment blueprint was agreed and a substantial bank of 1,127 English language progress test items was successfully calibrated after pilot testing to form the computer adaptive progress test (CA-PT) item bank. Results from the CA-PT pilot showed reliable convergence to stable estimates of ability, low standard errors of measurement and high test reliability for all participants. This study shows that an international collaborative consortium approach enables effective development of progress testing resources appropriate for computer adaptive testing, with potential for application across international borders and in populations where English is not the native language. Pooling resources internationally facilitates the comparison and development of appropriate assessment blueprints and the efficient generation of high-quality assessment items.
AB - Longitudinal progress testing promotes self-directed deep learning across a full spectrum of knowledge, enabling early detection of underperformance and opportunities for remediation. Computer adaptive testing (CAT), where the difficulty of a test dynamically adjusts according to a test taker's ability, has benefits in a progress testing context, but significant resource and experience is required to develop appropriate test materials. This study describes how a transnational consortium from eight medical schools in five countries across Europe was formed to develop a computer adaptive progress test applicable across international curricula. 1,212 students from more than 40 nationalities took part in the study, of whom more than 70% were not native English speakers, though nearly all reported competence in English. A content map for an international assessment blueprint was agreed and a substantial bank of 1,127 English language progress test items was successfully calibrated after pilot testing to form the computer adaptive progress test (CA-PT) item bank. Results from the CA-PT pilot showed reliable convergence to stable estimates of ability, low standard errors of measurement and high test reliability for all participants. This study shows that an international collaborative consortium approach enables effective development of progress testing resources appropriate for computer adaptive testing, with potential for application across international borders and in populations where English is not the native language. Pooling resources internationally facilitates the comparison and development of appropriate assessment blueprints and the efficient generation of high-quality assessment items.
U2 - 10.1016/j.caeai.2022.100083
DO - 10.1016/j.caeai.2022.100083
M3 - Article
SN - 2666-920X
VL - 3
JO - Computers and Education: Artificial Intelligence
JF - Computers and Education: Artificial Intelligence
M1 - 100083
ER -