Association for Public Policy Analysis & Management

Recently, a Tennessee policy went into effect introducing discontinuities in the number of formal observations assigned to teachers based on their prior level of effectiveness (LOE), a continuous measure determined by observational ratings, student growth and student achievement measures. (Tennessee State Board of Education, 2013). After each formal observation, observers are to provide feedback so teachers may use that information to improve their performance (Alexander, 2016). Given the importance of teacher effectiveness to short- and long-run student outcomes (Chetty, Friedman, & Rockoff, 2014), and increase in annual observations by many states since the early 2010s (Kraft & Gilmour, 2016), it is important to understand how policy-induced changes in annual observations impact teacher effectiveness.

The purpose of this study is to identify the effect of the marginal, policy-induced annual observation on subsequent improvements in end-of-year observational ratings and value-added measures of tested teachers.

Sample

Data come from 2012-13 through 2014-15 TDOE administrative datasets. Administrative data contain teacher observational ratings, value-added scores, observation dates, and demographics.

Research Design

Discontinuities in the policy-based assignment of annual observations beckons a regression discontinuity design (RDD), however, there is substantial non-compliance with this assignment. I instrument for the number of observations actually received using exogenous variation in LOE just on either side of the LOE thresholds where discontinuities in annual observations should exist according to policy (i.e. local linear RDD). While observers could attempt to manipulate a teacher’s observation rating, at least 40% of a teacher’s LOE is determined by student test scores.

Preliminary Findings

One more policy-induced observation per year lowers summative observational ratings across various identification strategies. Using some identification strategies value-added is lowered, otherwise it is unaffected. Under no circumstances is there evidence that the local average treatment effect (LATE) improves teacher performance.

Subsequent analyses will explore if different operationalizations of “formal observations” alter estimated effects, the LATE on within-year growth in observations scores (i.e. difference between last and first score received), and if the efficacy of policy-induced formal observations depends on rater effectiveness. More than 80% of raters are school administrators, who receive their own LOE.

Alexander, K. (2016). TEAM Evaluator Training. Nashville, TN: Tennessee Department of Education.

Chetty, R., Friedman, J. N., & Rockoff, J. E. (2014). The Long Term Impacts of Teachers: Teacher Value Added and Student Outcomes in Adulthood. American Economic Review, 104(9), 2633–2679.

Kraft, M. A., & Gilmour, A. F. (2016). Revisiting the Widget Effect: Teacher Evaluation Reforms and the Distribution of Teacher Effectiveness. In Association of Education Finance and Policy (pp. 1–31). Washington, D.C.

Tennessee State Board of Education. Teacher and Principal Evaluation Policy (2013). Nashville, TN.