Association for Public Policy Analysis & Management

People’s ability to find, keep, and advance in a job depends on self-regulation skills in addition to education, work experience, and technical skills (Almlund et al. 2011). Self-regulation skills include the ability to finish tasks, stay organized, and intentionally control emotions and behaviors. Research has shown that these skills are important in attaining goals and in determining life outcomes, including those related to employment (Almlund et al. 2011). Research has also shown that interventions can both strengthen self-regulation skills and encourage their use (Kautz et al. 2014).

In response to this research, some employment programs, including those offered as part of the Temporary Assistance for Needy Families (TANF) program, use strategies designed to strengthen and boost participants’ use of self-regulation skills (Cavadel et al. 2016; Kautz et al. 2014). To assess the effectiveness of these strategies, evaluators need a way to measure self-regulation skills. This presentation will discuss challenges in measuring self-regulation skills in evaluations of employment programs for low-income populations and provide guidance on selecting measures in this context.

Measuring self-regulation skills is challenging in any context because, unlike physical characteristics, self-regulation skills cannot be directly observed. Instead, self-regulation skills are always measured using behaviors broadly defined—how people act in various situations or perform on certain tasks (Heckman and Kautz 2012). Such behaviors are typically collected through one of four modes: (1) a self-report in a survey or interview that asks people about how they tend to behave; (2) an observer report in a survey or interview; (3) a performance task designed to capture particular self-regulation skills; or (4) administrative records about behaviors, such as attendance.

Four key challenges arise when applying these existing approaches to evaluations of employment programs for low-income populations. First, since measures of self-regulation skills are based on behaviors, they can reflect aspects of a person’s situation (for example, their background or financial resources) in addition to their skills. Second, most existing measures were developed for purposes other than program evaluation, such as describing characteristics of populations generally or for diagnosing psychological problems. Third, most existing measures were not designed for use in low-income populations. Fourth, some measurement modes—such as performance tasks—take a long time to administer or require special technology.

To address these challenges, we suggest that measures of self-regulation should: (1) relate to employment outcomes of interest; (2) capture skills that could be influenced by the program; (3) account for confounding factors presented by the person’s situation that affect measurement but not skills; and (4) be feasible to administer in the setting of an evaluation of an employment program. To meet these criteria, we suggest using a set of both general measures of self-regulation as well as ones that are specific to the employment context, collecting information on other aspects of the participants’ situations that can be affected by the program, modifying measures to fit the target population, and conducting analyses to assess the reliability and validity of selected measures.