Association for Public Policy Analysis & Management

Recent federally-supported K-12 school turnaround efforts have specified school performance thresholds below which schools are eligible for intervention. These strict assignments to treatment on prior performance composite create opportunities to estimate a local average treatment effect (LATE) at the threshold between treatment and control using a regression discontinuity design (RDD). Given the strong internal validity and causal warrant of RDD, a number of turnaround evaluations have employed these designs. However, the causal warrant of LATE estimates is most relevant to cases in which program expansion is considered at the margin, while the generalization of the LATE as an average treatment effect (ATE) rests on an assumption of homogenous treatment effects. We thus argue that where policymakers are weighing the potential adoption or continuation of turnaround efforts, rather than marginal program expansions, RDD results may not reflect the more policy-relevant effects of turnaround on average or for the lowest-performing of low-performing schools most in need of support services. To illustrate, this study contrasts estimates of RDD, difference-in-difference (DD), and comparative interrupted time series (CITS) models of the impact of North Carolina’s school turnaround efforts on student academic outcomes in North Carolina’s lowest-performing schools. Specifically, we seek to determine whether the negative LATE estimates from the RDD and positive average treatment effect (ATE) estimates from the DD represent valid answers to separate questions.

As background, the District & School Transformation (DST) division of the North Carolina Department of Public Instruction has used a share of the state’s $400M Race to the Top grant funds to support its Turning Around Low-Achieving Schools (TALAS) initiative. Designation for school turnaround followed strict assignment criteria with 2009-2010 performance composites (pass rate for end-of-grade and end-of-course exams) determining which schools represented the bottom five percent in the state. Treatment schools received services from 2011-2012 through 2014-2015. Federal turnaround models adopted by the state required, among other reforms: leadership change, staff replacement, increased recruitment and retention efforts, and coaching-intensive professional development.

Both the RDD and DD models show positive effects on student outcomes in secondary schools, and results are also positive in elementary and middle schools under the DD. The RDD finds negative effects for elementary and middle schools near the threshold, creating a contrast between the LATE and ATE estimates for K-8 student outcomes. Focusing on these results, we subject both the RDD and DD to a number of robustness checks and use a CITS design for a second ATE estimate to help determine whether the conflicting results represent valid answers to separate questions. If the LATE and ATE are each valid but offer conflicting estimates, the results would be consistent with heterogeneous treatment effects and suggest that while TALAS had an overall positive effect on student academic outcomes in low-achieving schools, the program was too inclusive such that students in treatment schools near the threshold may have fared better in the absence of intervention. Discussion focuses on implications for both the design and dissemination of research on school turnaround.