When the Late Ain't ATE: Investigating and Interpreting Differences in School Turnaround Effect Estimates
*Names in bold indicate Presenter
As background, the District & School Transformation (DST) division of the North Carolina Department of Public Instruction has used a share of the state’s $400M Race to the Top grant funds to support its Turning Around Low-Achieving Schools (TALAS) initiative. Designation for school turnaround followed strict assignment criteria with 2009-2010 performance composites (pass rate for end-of-grade and end-of-course exams) determining which schools represented the bottom five percent in the state. Treatment schools received services from 2011-2012 through 2014-2015. Federal turnaround models adopted by the state required, among other reforms: leadership change, staff replacement, increased recruitment and retention efforts, and coaching-intensive professional development.
Both the RDD and DD models show positive effects on student outcomes in secondary schools, and results are also positive in elementary and middle schools under the DD. The RDD finds negative effects for elementary and middle schools near the threshold, creating a contrast between the LATE and ATE estimates for K-8 student outcomes. Focusing on these results, we subject both the RDD and DD to a number of robustness checks and use a CITS design for a second ATE estimate to help determine whether the conflicting results represent valid answers to separate questions. If the LATE and ATE are each valid but offer conflicting estimates, the results would be consistent with heterogeneous treatment effects and suggest that while TALAS had an overall positive effect on student academic outcomes in low-achieving schools, the program was too inclusive such that students in treatment schools near the threshold may have fared better in the absence of intervention. Discussion focuses on implications for both the design and dissemination of research on school turnaround.