Skip to main navigation Skip to search Skip to main content

Sample size determination for hypothesis testing on the intraclass correlation coefficient in a two-way analysis of variance model

Research output: Contribution to journal › Article › Academic › peer-review

Abstract

Reliability evaluation is critical in fields such as psychology and medicine to ensure accurate diagnosis and effective treatment management. When participants are evaluated by the same raters, a two-way ANOVA model is suitable to model the data, with the intraclass correlation coefficient (ICC) serving as the reliability metric. In these domains, the ICC for agreement (ICCa) is commonly used, as the values of the measurements themselves are of interest. Designing such reliability studies requires determining the sample size of participants and raters for the ICCa. Although procedures for sample size determination exist based on the expected width of the confidence interval for the ICCa, there is limited work on hypothesis testing. This paper addresses this gap by proposing procedures to ensure sufficient power to statistically test whether the ICCa exceeds a predetermined value, utilizing confidence intervals for the ICCa. We compared the available confidence interval methods for the ICCa and proposed sample size procedures using the lower confidence limit of the best performing methods. These procedures were evaluated considering the empirical power of the hypothesis test under various parameter configurations. Furthermore, these procedures are implemented in an interactive R shiny app, freely available to researchers for determining sample sizes.
Original languageEnglish
Pages (from-to)590-620
Number of pages31
JournalBritish Journal of Mathematical & Statistical Psychology
Volume79
Issue number3
Early online date2025
DOIs
Publication statusPublished - Nov 2026

Keywords

  • Interrater reliability
  • Measurement errors
  • Observer variation
  • Power
  • Reproducibility of results
  • Sample size calculation
  • APPROXIMATE INTERVAL ESTIMATION
  • CONFIDENCE-INTERVALS
  • LINEAR-COMBINATIONS
  • PRECISION
  • NUMBER

Fingerprint

Dive into the research topics of 'Sample size determination for hypothesis testing on the intraclass correlation coefficient in a two-way analysis of variance model'. Together they form a unique fingerprint.

Cite this