Association for Public Policy Analysis & Management

Authors:

Melinda Adnot, University of Virginia; Tom Dee, Stanford University; Veronica Katz, University of Virginia; Jim Wyckoff, University of Virginia.

Presenter: Aliza Husain, University of Virginia.

Background:

Although there is a long history of performance pay in education, most of these initiatives have not been sustained for long and have generally been viewed as unsuccessful (Murnane and Cohen, 1986). For instance, evaluation of merit-pay experiments in Nashville, New York, and Chicago found that teacher performance did not increase as a result of performance pay incentives (Fryer, 2013; Glazerman and Seifullah, 2012; Springer et al., 2010). In contrast to these null findings, Dee and Wyckoff (2013) present causal evidence that IMPACT, the teacher evaluation system implemented in D.C. Public Schools (DCPS) in 2009, improved the performance of teachers confronting strong dismissal and financial incentives. However, Dee and Wyckoff did not examine the relationship between IMPACT-induced improvements in teacher performance and student achievement. Forthcoming work by Adnot, Dee, Katz, and Wyckoff that builds on these prior findings provides new evidence that highly incentivized teachers at the low end of the performance distribution are able to respond to incentives and change their overall job performance, as well as their impact on student learning outcomes in the next year.

To assess the effect of IMPACT and associated reforms on student achievement, we will compare changes in DCPS student achievement before and after the implementation of IMPACT to changes in student achievement in other school districts over the same period of time. The validity of this analysis hinges on the selection of a credible counterfactual. To this end, we will compare the pre/post-reform changes in DCPS to the contemporaneous changes observed in: (1) student-level data from DCPS and DC charter schools, (2) school-level data for DCPS and Baltimore City and Prince Georges County public schools in Maryland, and (3) district-level NAEP data available over several years for DCPS and other districts participating in the Trial Urban District Assessments (TUDA). Collectively, these comparison groups will provide a robust assessment of the effects of IMPACT and associated reforms on student achievement.

Research Question:

1. How has test-based performance changed in DCPS before and after the implementation of IMPACT? How do these changes compare to the differences observed in other comparison school districts over the same period of time?

Method:

Using a difference-in-difference framework, the basic analytic model is

Y_it = a + b₀POST + b₁DCPS + b₂DCPS*POST +bX_it + y_i +e_it

where Y_it is a measure of student achievement for unit i in year t, a and b₀ capture baseline and post-IMPACT student achievement for the comparison group, respectively, b₁ and b₂ capture baseline and post-IMPACT student achievement for DCPS, respectively, X_it is a vector of time-varying student socio-demographics known to be associated with student achievement, and g_i is a unit-level fixed effect. The coefficient of interest is b₂ as it indicates changes in post-IMPACT student achievement in DCPS, net of contemporaneous changes in student achievement in the comparison group.