Assessing College Readiness: Internal Structure Validity Evidence for Multistage Diagnostic Reading Assessment

April 13, 2023, Chicago, IL

Oxana Rosca, Kimberly Colvin, Heidi Andrade, Jason Bryer

Abstract Using data from 2,190 newly enrolled college students who completed the DAACS Self-Regulated Learning (SRL) Survey at the University at Albany and the University of Maryland Global Campus, we examined internal consistency estimates for SRL scale scores (multidimensional) and subscale scores (primarily unidimensional). The 66-item SRL Survey includes four scales, Metacognition, Motivation, Using Learning Strategies, and Self-Efficacy, and 14 subscales.

We compared Cronbach’s alpha (α), Revelle’s beta (β), and McDonald’s omega (hierarchical ωh and total ωt), and evaluated key psychometric conditions that influence their interpretation, including dimensionality, the strength of a general factor, and essential τequivalency. Several SRL subscales demonstrated moderate-to-high internal consistency across indices, with ωt typically yielding the largest estimates, consistent with expectations for multidimensional constructs.

Overall, the findings provide evidence for the consistency of item responses within SRL scales and subscales and highlight the importance of selecting reliability coefficients that align with the factor structure of the DAACS SRL Survey, particularly in the presence of multidimensionality and violated assumptions underlying α.

What this means for users DAACS SRL scores can be interpreted as reasonably consistent indicators of students’ self-regulated learning skills, with reliability supported by coefficients that appropriately reflect each scale’s psychometric structure.

View the paper here https://www.researchgate.net/publication/370004535_Reliability_Coefficients_for_a_Single-administration_Linear_Assessment