An R Package for Optimizing the Composite Reliability in Multivariate Nested Designs

The reliability of assessment tools is critical for accurately monitoring student performance in various educational contexts. When multiple assessments are combined to form an overall evaluation, each assessment serves as a data point contributing to the student's performance within a broader educational framework. Determining composite reliability in such cases can be complex, particularly in naturalistic, unbalanced datasets in nested designs, which are common in programmatic and workplace-based assessments, where students are evaluated on unique, practical occasions. This paper introduces the compositeReliabilityInNestedDesigns package in R, designed to estimate composite reliability using multivariate generalizability theory and enhance the analysis of assessment data. The package produces extensive Generalizability and Decision study results with graphical interpretations. Composite reliability incorporates weights and covariance to integrate results across assessment tools. Weights can be optimized to minimize standard error of measurement or maximize reliability. Overall, the package's flexible use and optimization empowers assessment tailoring and robust insights into student performance. The approach is suitable for programmatic assessment. The package facilitates reliable, comprehensive evaluation across diverse assessments.