Asymptotic variability of (multilevel) multirater kappa coefficients

Sophie Vanbelle*

*Corresponding author for this work

Research output: Contribution to journalArticleAcademicpeer-review


Agreement studies are of paramount importance in various scientific domains. When several observers classify objects on categorical scales, agreement can be quantified through multirater kappa coefficients. In most statistical packages, the standard error of these coefficients is only available under the null hypothesis that the coefficient is equal to zero, preventing the construction of confidence intervals in the general case. The aim of this paper is triple. First, simple analytic formulae for the standard error of multirater kappa coefficients will be given in the general case. Second, these formulae will be extended to the case of multilevel data structures. The formulae are based on simple matrix algebra and are implemented in the R package multiagree. Third, guidelines on the choice between the different mulitrater kappa coefficients will be provided.

Original languageEnglish
Pages (from-to)3012-3026
Number of pages15
JournalStatistical Methods in Medical Research
Issue number10-11
Publication statusPublished - Nov 2019


  • Fleiss's kappa
  • Conger kappa
  • pairwise agreement
  • hierarchical
  • nested
  • rater


Dive into the research topics of 'Asymptotic variability of (multilevel) multirater kappa coefficients'. Together they form a unique fingerprint.

Cite this