We look forward to seeing you at our upcoming presentations

LOS ANGELES, WEDNESDAY, APRIL 8, TO SUNDAY, APRIL 12, 2026

                        American Educational Research Association (AERA)

Free will as an obstacle to randomized controlled trials

Saturday, April 11th,Los Angeles

Jason Bryer, Angela M. Lui,Heidi L. Andrade,Piet Wesling

Abstract: Randomized controlled trials (RCTs) are considered the gold standard for estimating causality. However, conducting research in educational settings using RCTs can result in unnatural intervention implementations due to participant agency — that is, their free will. This study explores the prevalence of RCTs in education using What Works Clearinghouse (WWC) data and examines challenges from implementing seven RCTs across five institutions. We found that RCTs were less likely than non-RCTs to report positive effects and faced issues of noncompliance. Propensity score analysis (PSA) was used to estimate treatment effects and revealed significant differences not evident in intent-to-treat analysis. These findings suggest that free will complicates RCT assumptions and that PSA may better capture the heterogeneous effects of educational interventions.

College students’ responses to feedback from a diagnostic assessment of self-regulated learning

Thursday, April 9th, Los Angeles

Piet Wesling, Angela M. Lui, Joseph Garofalo, Diana Akhmedjanova, Heidi L. Andrade, Jason Bryer

Abstract:This study explores college students’ cognitive, emotional, and behavioral responses to feedback from an online diagnostic assessment of self-regulated learning skills. Using qualitative methods and a think-aloud protocol, we examined how seven students interpreted and reacted to personalized feedback, including descriptive text, dots as scores, and videos on suggested strategies. Findings revealed that students engaged cognitively by making judgments and attributions. Meaning-making frequently involved linking feedback to personal experiences. Emotional responses varied from appreciation to overwhelm. Few students made concrete decisions about next steps. Results underscore the complex interplay of interpretation, emotion, and motivation in feedback processing. Implications are discussed.

                        The National Council on Measurement in Education (NCME)

Students‘ responses to feedback on a diagnostic assessment of self-regulated learning – A theory of action

Saturday, April 11th, Los Angeles

Piet Wesling, Angela M. Lui, Joseph Garofalo, Diana Akhmedjanova, Heidi L. Andrade

Abstract: The purpose of this study is to assess validity evidence for the formative feedback of a diagnostic assessment of self-regulated learning (SRL) skills. Because the diagnostic assessment system serves both measurement and intervention functions, we adopt a theory of action (ToA) approach to evaluate the alignment between key elements of the feedback system, including test components, action mechanisms, and intended and unintended impacts. Focusing specifically on the SRL feedback, we articulate the ToA for feedback use and examine interpretive claims regarding how students engage with, interpret, and respond to the feedback. Data sources include assessment system documentation, prior project publications, an open-ended survey of system developers, and think-aloud protocols and semi-structured retrospective interviews with undergraduate students (n = 7). Students completed the SRL survey and then reviewed personalized feedback while verbalizing their cognitive and affective responses. Using inductive coding, we analyzed students’ engagement with feedback to evaluate the extent to which empirical evidence supports the proposed interpretive claims and to identify important unintended consequences. Findings provide validity evidence based on feedback response processes and illuminate how learners make sense of and act on diagnostic SRL feedback. The study demonstrates the value of ToA as a framework for extending validity research beyond test items and scores to include feedback use and consequences, offering practical implications for both assessment design and educational validity theory.

Bootstrapping to determine the optimal number of clusters

Los Angeles

Jason Bryer