National Council on Measurement in Education

By DAACS | April 5, 2019

Relationship Between Intraclass Correlation and Percent Rater Agreement

Authors: Jason Bryer; Guher Gorgun


Inter-rater reliability (IRR) is a critical component of establishing the reliability of measures when more than one rater is necessary. There are numerous IRR statistics available to researchers including percent rater agreement, Cohen’s Kappa, and several types of intraclass correlations (ICC). Several methodologists suggest using ICC over percent rater agreement (Hallgren, 2012; Koo & Li, 2016; McGraw & Wong, 1996; Shrout & Fleiss, 1979). However, the literature provides little guidance on the interpretation of ICC results. This article explores the relationship between ICC and percent rater agreement using simulations. Results suggest that ICC and percent rater agreement are highly correlated ($R^2 > 0.9$) for most designs.

Validity and Reliability of the DAACS Writing Assessment

Authors: Diana Akhmedjanova; Angela M. Lui; Heidi L. Andrade; Jason Bryer


The DAACS system assesses incoming college students’ readiness in reading, writing, mathematics, and self-regulated learning. The purpose of this study is to examine validity and reliability evidence for the internal structure of the writing assessment. The evidence suggests that our conceptual framework holds for both human and machine scored essays.