Association for Public Policy Analysis & Management

Panel: Measuring Soft Skills for Evaluations and Policy: Challenges and Innovations

(Methods and Tools of Analysis)

Recent research has demonstrated that success in life requires not only knowledge and intelligence but soft skills, such as persistence, self-control, and emotion regulation. For many important outcomes, such as educational attainment, health, earnings, and employment, the predictive power of soft skills is similar to or higher than that of available measures of intelligence (Heckman & Kautz, 2012). Perhaps most promising for public policy is that soft skills appear to be malleable throughout young adulthood and can be developed through interventions (Kautz et al., 2014; Roberts & Jackson, 2008). Given this research, evaluators and policymakers have been searching for reliable and low-burden measures of soft skills to inform policy and practice (West et al. 2016). Progress has been slow, however, in part because of a pervasive view that soft skills are difficult to measure reliably in this context.

This proposed panel will consist of presentations from four interdisciplinary papers that (1) outline the challenges of measuring soft skills for use in evaluations and policy and (2) discuss promising and innovative approaches that address these challenges. They will answer four key questions:

1. What are the challenges in measuring soft skills for use in evaluations? The first presentation will discuss both technical and practical challenges in measuring self-regulation skills (soft skills) in evaluations, focusing on employment programs for low-income populations. It will also offer some recommendations to address the challenges.
2. What are potential limitations of standard, self-reported measures of soft skills and what are alternative approaches? The second paper will elaborate further on limitations of commonly-used measures of soft skills and will present alternative, innovative approaches including forced-choice methods, anchoring vignettes, situational judgement tests, and performance tasks.
3. To what extent can behaviors—such as those collected by schools—be used to measure soft skills? The third presentation will discuss another alternative to commonly-used measures. It will provide results on how behaviors collected in school administrative data can serve as an “unobtrusive measure” of students’ ability to control their behavior, a key soft skill typically tracked via student surveys.
4. Can the way in which people respond to surveys and take tests serve as a proxy for soft skills? The fourth presentation will discuss how soft skills can be measured using information about how people complete surveys and tests, including how long they spend on items, whether they skip items, or provide inconsistent answers. This approach is a promising low-burden way to inform practice and policy.

This panel fits with the conference theme of encouraging innovation and improvement, because it presents innovative methods for measuring soft skills for evaluation and policy, an area for which collaboration between researchers, policy makers, and program leaders is essential. The presentation panel consists of researchers from diverse fields including economics, psychology, and education. The proposed chair has experience evaluating programs designed to boost soft skills. The discussant is a practitioner who works closely with soft skill development programs, so will provide valuable insights into the feasibility of the methods.