Abstract
Reliability evaluation is critical in fields such as psychology and medicine to ensure accurate diagnosis and effective treatment management. When participants are evaluated by the same raters, a two-way ANOVA model is suitable to model the data, with the intraclass correlation coefficient (ICC) serving as the reliability metric. In these domains, the ICC for agreement (ICCa) is commonly used, as the values of the measurements themselves are of interest. Designing such reliability studies requires determining the sample size of participants and raters for the ICCa. Although procedures for sample size determination exist based on the expected width of the confidence interval for the ICCa, there is limited work on hypothesis testing. This paper addresses this gap by proposing procedures to ensure sufficient power to statistically test whether the ICCa exceeds a predetermined value, utilizing confidence intervals for the ICCa. We compared the available confidence interval methods for the ICCa and proposed sample size procedures using the lower confidence limit of the best performing methods. These procedures were evaluated considering the empirical power of the hypothesis test under various parameter configurations. Furthermore, these procedures are implemented in an interactive R shiny app, freely available to researchers for determining sample sizes.
| Original language | English |
|---|---|
| Pages (from-to) | 590-620 |
| Number of pages | 31 |
| Journal | British Journal of Mathematical & Statistical Psychology |
| Volume | 79 |
| Issue number | 3 |
| Early online date | 2025 |
| DOIs | |
| Publication status | Published - Nov 2026 |
Keywords
- Interrater reliability
- Measurement errors
- Observer variation
- Power
- Reproducibility of results
- Sample size calculation
- APPROXIMATE INTERVAL ESTIMATION
- CONFIDENCE-INTERVALS
- LINEAR-COMBINATIONS
- PRECISION
- NUMBER
Fingerprint
Dive into the research topics of 'Sample size determination for hypothesis testing on the intraclass correlation coefficient in a two-way analysis of variance model'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver