Association for Public Policy Analysis & Management

In their efforts to improve the quality of the teacher workforce, researchers and policymakers have sought to identify “multiple measures” of effectiveness that capture the range of skills that teachers bring to the classroom (Kane & Staiger, 2012). Over the last several years, debates around which measures are most appropriate and least prone to bias have focused predominantly on teachers’ contribution to student test scores – often referred to as “value added” – and observations of teaching practice (Chetty, Freidman, & Rockoff, 2014; Kane, McCaffrey, Miller, & Staiger, 2013; Rothstein, 2014; Whitehurst, Chingos, & Lindquist, 2014). However, as of 2016, over 40 states also required or recommended use of a third metric. While many leave the choice of which additional measure(s) to use open to individual districts, 36 states list student surveys or “non-cognitive” school behaviors as possible sources of data (Center of Great Teachers and Leaders, 2013).

The intent to evaluate teachers against a range of “non-cogntive” outcomes has some substantive backing. Measures including students’ behavior, self-efficacy, happiness, and motivation have been linked to long-term outcomes in the labor market and beyond (Chetty et al., 2011; Heckman, & Rubinstein, 2001; Lindqvist & Vestman, 2011; Mueller & Plug, 2006). Further, teachers appear to impact these outcomes in ways that are only weakly related to their effect on students’ test scores (Blazar & Kraft, 2015; Jenning & DiPrete, 2010; Kraft & Grace, 2016). At the same time, some in the research community caution that these sorts of metrics may be prone to substantial biases, which would make them unsuitable for evaluation and improvement efforts (Duckworth & Yeager, 2015).

I examine bias in teacher effects on students’ attitudes and behaviors by drawing on a unique dataset from the National Center for Teacher Effectiveness in which teachers were randomly assigned to class rosters within schools. Participating students completed a survey with items that asked about their self-efficacy in math, happiness in class, and behavior in class. These data allowed me to test whether non-experimental estimates of teachers’ effectiveness at improving these attitudes and behaviors predicted these same outcomes following random assignment. In other words, I ask: Do non-experimental teacher effects on students’ attitudes and behaviors reflect causal relationships?

Findings indicate that the degree of bias varies widely across outcome measures used to calculate teacher effect estimates. For teacher effects on students’ self-reported behavior in class, non-experimental or predicted differences come close to actual differences in student outcomes following the random assignment of teachers to classes, thus likely reflecting a causal relationship. Teacher effects on students’ self-efficacy in math have moderate predictive validity. However, even the best models likely contain at least 46% bias. Finally, I find that teacher effects on students’ happiness in class have no predictive validity. As the first random assignment study to focus specifically on teacher effects on students’ attitudes and behaviors, these findings can serve as a benchmark for future work and contribute to a growing body of evidence validating measures of teacher effectiveness.