By DAACS | April 5, 2019
Relationship Between Intraclass Correlation and Percent Rater Agreement
Authors: Jason Bryer; Guher Gorgun
Abstract
Inter-rater reliability (IRR) is a critical component of establishing the reliability of measures when more than one rater is necessary. There are numerous IRR statistics available to researchers including percent rater agreement, Cohen’s Kappa, and several types of intraclass correlations (ICC). Several methodologists suggest using ICC over percent rater agreement (Hallgren, 2012; Koo & Li, 2016; McGraw & Wong, 1996; Shrout & Fleiss, 1979). However, the literature provides little guidance on the interpretation of ICC results. This article explores the relationship between ICC and percent rater agreement using simulations. Results suggest that ICC and percent rater agreement are highly correlated ($R^2 > 0.9$) for most designs.
Validity and Reliability of the DAACS Writing Assessment
Authors: Diana Akhmedjanova; Angela M. Lui; Heidi L. Andrade; Jason Bryer
Abstract
The DAACS system assesses incoming college students’ readiness in reading, writing, mathematics, and self-regulated learning. The purpose of this study is to examine validity and reliability evidence for the internal structure of the writing assessment. The evidence suggests that our conceptual framework holds for both human and machine scored essays.