Automatic conversational assessment using large language model technology

Jan Bergerhoff; Johannes Bendler; Stefan Stefanov; Enrico Cavinato; Leonard Esser; Tommy Tran; Aki Härmä

doi:10.1145/3702163.3702169

Automatic conversational assessment using large language model technology

Jan Bergerhoff, Johannes Bendler, Stefan Stefanov, Enrico Cavinato, Leonard Esser, Tommy Tran, Aki Härmä

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

Abstract

Student evaluation is an important, yet costly, part of instruction. Traditional exams are a burden for teachers and stressful for students. This paper uses a large language model (LLM) technology to create a system for Automated Conversational Assessment, ACA, where a dialog system, based on content and intended learning outcomes, interviews the student to determine the level of learning. In a pilot experiment in a university course, we found that the ACA system scores correlate with the grades given by a human and also have a positive correlation with the results of a conventional exam of the same students. Based on a questionnaire study, the students responded that the assessment was perceived to be fair and acceptable.

Original language	Undefined/Unknown
Title of host publication	Proceedings of the 2024 16th International Conference on Education Technology and Computers
Place of Publication	New York, NY, USA
Publisher	Association for Computing Machinery
Pages	39-45
Number of pages	7
ISBN (Print)	9798400717819
DOIs	https://doi.org/10.1145/3702163.3702169
Publication status	Published - 21 Jan 2025

Publication series

Series	Proceedings of the International Conference on Education Technology and Computers

Access to Document

10.1145/3702163.3702169

https://doi.org/10.1145/3702163.3702169

Cite this

Bergerhoff, Jan ; Bendler, Johannes ; Stefanov, Stefan et al. / Automatic conversational assessment using large language model technology. Proceedings of the 2024 16th International Conference on Education Technology and Computers. New York, NY, USA : Association for Computing Machinery, 2025. pp. 39-45 (Proceedings of the International Conference on Education Technology and Computers).

@inproceedings{60bab9f01f8f455f962c0323faa87074,

title = "Automatic conversational assessment using large language model technology",

abstract = "Student evaluation is an important, yet costly, part of instruction. Traditional exams are a burden for teachers and stressful for students. This paper uses a large language model (LLM) technology to create a system for Automated Conversational Assessment, ACA, where a dialog system, based on content and intended learning outcomes, interviews the student to determine the level of learning. In a pilot experiment in a university course, we found that the ACA system scores correlate with the grades given by a human and also have a positive correlation with the results of a conventional exam of the same students. Based on a questionnaire study, the students responded that the assessment was perceived to be fair and acceptable.",

author = "Jan Bergerhoff and Johannes Bendler and Stefan Stefanov and Enrico Cavinato and Leonard Esser and Tommy Tran and Aki H{\"a}rm{\"a}",

year = "2025",

month = jan,

day = "21",

doi = "10.1145/3702163.3702169",

language = "Undefined/Unknown",

isbn = "9798400717819",

series = "Proceedings of the International Conference on Education Technology and Computers",

publisher = "Association for Computing Machinery",

pages = "39--45",

booktitle = "Proceedings of the 2024 16th International Conference on Education Technology and Computers",

address = "United States",

}

Bergerhoff, J, Bendler, J, Stefanov, S, Cavinato, E, Esser, L, Tran, T & Härmä, A 2025, Automatic conversational assessment using large language model technology. in Proceedings of the 2024 16th International Conference on Education Technology and Computers. Association for Computing Machinery, New York, NY, USA, Proceedings of the International Conference on Education Technology and Computers, pp. 39-45. https://doi.org/10.1145/3702163.3702169

Automatic conversational assessment using large language model technology. / Bergerhoff, Jan; Bendler, Johannes; Stefanov, Stefan et al.
Proceedings of the 2024 16th International Conference on Education Technology and Computers. New York, NY, USA: Association for Computing Machinery, 2025. p. 39-45 (Proceedings of the International Conference on Education Technology and Computers).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - Automatic conversational assessment using large language model technology

AU - Bergerhoff, Jan

AU - Bendler, Johannes

AU - Stefanov, Stefan

AU - Cavinato, Enrico

AU - Esser, Leonard

AU - Tran, Tommy

AU - Härmä, Aki

PY - 2025/1/21

Y1 - 2025/1/21

N2 - Student evaluation is an important, yet costly, part of instruction. Traditional exams are a burden for teachers and stressful for students. This paper uses a large language model (LLM) technology to create a system for Automated Conversational Assessment, ACA, where a dialog system, based on content and intended learning outcomes, interviews the student to determine the level of learning. In a pilot experiment in a university course, we found that the ACA system scores correlate with the grades given by a human and also have a positive correlation with the results of a conventional exam of the same students. Based on a questionnaire study, the students responded that the assessment was perceived to be fair and acceptable.

AB - Student evaluation is an important, yet costly, part of instruction. Traditional exams are a burden for teachers and stressful for students. This paper uses a large language model (LLM) technology to create a system for Automated Conversational Assessment, ACA, where a dialog system, based on content and intended learning outcomes, interviews the student to determine the level of learning. In a pilot experiment in a university course, we found that the ACA system scores correlate with the grades given by a human and also have a positive correlation with the results of a conventional exam of the same students. Based on a questionnaire study, the students responded that the assessment was perceived to be fair and acceptable.

U2 - 10.1145/3702163.3702169

DO - 10.1145/3702163.3702169

M3 - Conference article in proceeding

SN - 9798400717819

T3 - Proceedings of the International Conference on Education Technology and Computers

SP - 39

EP - 45

BT - Proceedings of the 2024 16th International Conference on Education Technology and Computers

PB - Association for Computing Machinery

CY - New York, NY, USA

ER -

Bergerhoff J, Bendler J, Stefanov S, Cavinato E, Esser L, Tran T et al. Automatic conversational assessment using large language model technology. In Proceedings of the 2024 16th International Conference on Education Technology and Computers. New York, NY, USA: Association for Computing Machinery. 2025. p. 39-45. (Proceedings of the International Conference on Education Technology and Computers). doi: 10.1145/3702163.3702169