Modeling agreement on categorical scales in the presence of random scorers

S. Vanbelle, E. Lesaffre

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Kappa coefficients are often used to assess agreement between two fixed scorers on categorical scales. Cohen's version is popular for nominal scales and the weighted version for ordinal scales. In the present paper, similar agreement coefficients are defined for random scorers. A partial-Bayesian methodology is then developed to directly relate these agreement coefficients to predictors through a multilevel model. Statistical properties of the proposed approach are studied using simulations. Finally, the approach is applied to gynecological and medical imaging data.

Original languageEnglish
Pages (from-to)79-93
Number of pages15
JournalBiostatistics
Volume17
Issue number1
DOIs
Publication statusPublished - 1 Jan 2016

Keywords

  • Markov chain Monte Carlo
  • Multilevel
  • Nested
  • Rater
  • Reliability
  • WEIGHTED KAPPA
  • COEFFICIENT
  • RELIABILITY
  • PARAMETERS

Cite this

@article{70ed75fe1bf34afda8321c9b974abdbf,
title = "Modeling agreement on categorical scales in the presence of random scorers",
abstract = "Kappa coefficients are often used to assess agreement between two fixed scorers on categorical scales. Cohen's version is popular for nominal scales and the weighted version for ordinal scales. In the present paper, similar agreement coefficients are defined for random scorers. A partial-Bayesian methodology is then developed to directly relate these agreement coefficients to predictors through a multilevel model. Statistical properties of the proposed approach are studied using simulations. Finally, the approach is applied to gynecological and medical imaging data.",
keywords = "Markov chain Monte Carlo, Multilevel, Nested, Rater, Reliability, WEIGHTED KAPPA, COEFFICIENT, RELIABILITY, PARAMETERS",
author = "S. Vanbelle and E. Lesaffre",
year = "2016",
month = "1",
day = "1",
doi = "10.1093/biostatistics/kxv036",
language = "English",
volume = "17",
pages = "79--93",
journal = "Biostatistics",
issn = "1465-4644",
publisher = "Oxford University Press",
number = "1",

}

Modeling agreement on categorical scales in the presence of random scorers. / Vanbelle, S.; Lesaffre, E.

In: Biostatistics, Vol. 17, No. 1, 01.01.2016, p. 79-93.

Research output: Contribution to journalArticleAcademicpeer-review

TY - JOUR

T1 - Modeling agreement on categorical scales in the presence of random scorers

AU - Vanbelle, S.

AU - Lesaffre, E.

PY - 2016/1/1

Y1 - 2016/1/1

N2 - Kappa coefficients are often used to assess agreement between two fixed scorers on categorical scales. Cohen's version is popular for nominal scales and the weighted version for ordinal scales. In the present paper, similar agreement coefficients are defined for random scorers. A partial-Bayesian methodology is then developed to directly relate these agreement coefficients to predictors through a multilevel model. Statistical properties of the proposed approach are studied using simulations. Finally, the approach is applied to gynecological and medical imaging data.

AB - Kappa coefficients are often used to assess agreement between two fixed scorers on categorical scales. Cohen's version is popular for nominal scales and the weighted version for ordinal scales. In the present paper, similar agreement coefficients are defined for random scorers. A partial-Bayesian methodology is then developed to directly relate these agreement coefficients to predictors through a multilevel model. Statistical properties of the proposed approach are studied using simulations. Finally, the approach is applied to gynecological and medical imaging data.

KW - Markov chain Monte Carlo

KW - Multilevel

KW - Nested

KW - Rater

KW - Reliability

KW - WEIGHTED KAPPA

KW - COEFFICIENT

KW - RELIABILITY

KW - PARAMETERS

U2 - 10.1093/biostatistics/kxv036

DO - 10.1093/biostatistics/kxv036

M3 - Article

VL - 17

SP - 79

EP - 93

JO - Biostatistics

JF - Biostatistics

SN - 1465-4644

IS - 1

ER -