Association for Public Policy Analysis & Management

Executive function (EF) and regulation-related skills (RRS) are central to children’s early learning and social-emotional success (Blair & Razza, 2007; Ursache, Blair, & Raver, 2012). Methods for capturing EF and RRS range from computer-based tasks targeting specific skills (e.g., inhibitory control) to adult reports of broader behavior in context (e.g., behavioral regulation in the classroom). At present, however, there is no tool available that objectively captures children’s deployment of EF and RRS outside of highly controlled and emotionally neutral environments. The aim of this project is to fill this gap in the methodological toolkit for assessing EF and RRS. Specifically, we describe the development of a novel, ecologically valid observational tool for capturing children’s EF and RRS in early childhood classroom settings: the Regulation-Related Skills Measure (RRSM).

The RRSM was developed via a researcher-practitioner partnership using a multi-step process. First, we developed a conceptual framework based a comprehensive taxonomy of the literature that included core subdomains of EF and RRS (see “sub-domains” column of Table 1). Second, we developed 16 items (see “items” column of Table 1) and corresponding behavioral markers (see “behavioral markers” column of Table 1) that matched these sub-domains. Third, we conducted a pilot study assessing the performance of these items in relation to several known measures of EF and RRS, including the NIH Toolkit, teacher and parent reports using the Behavior Rating Inventory of Executive Function (BRIEF), and E4 wristbands that measure physiological signals of arousal and regulation, including skin conductance and heart-rate variability. The sample for the pilot study includes approximately 200 racially and socioeconomically diverse children from 18 preschool and kindergarten classrooms in Colorado and Massachusetts. Brief (approximately five-minute) video clips of children involved in day-to-day classroom activities were coded using the RRSM by one to three independent coders. Videos were coded using the RRSM on a four-point response scale, with higher scores indicating greater dysregulation.

Preliminary results of analyses focusing on 23 video clips from 15 children are shown in Table 2. The ICC of the RRSM total score was .76, indicating that there was more variation across than within children. Partial correlations adjusting for age and vocabulary skills suggest that children who showed more dysregulation on the RRSM performance lower on computer-based EF tasks (range r  = -.15 for the Flanker to -.39 for the Minnesota Executive Function Scale). Interestingly, children who showed more dysregulation on the RRSM showed lower dysregulation according to teacher and parent reports on the BRIEF (r = -.20 and -.63, respectively).

Results of this pilot study suggest that the RRSM may be a useful tool for capturing EF and RRS in the context of the everyday stressors. At the same time, additional work is needed to ensure the tool’s reliability, to reduce the proportion of “not observed” responses, and to understand counterintuitive correlations with adult reports. The final presentation will include results from the full pilot sample (including inter-rater reliability and correlations with physiological measures of reactivity), as well as implications for research and formative assessment.