A large urban Virginia district has engaged in a pay-for-performance teacher incentive policy awarded through the Teacher Inventive Fund (TIF) Grant competition. The policy provides a significant monetary award to teachers in eligible schools that qualify to receive the performance-based compensation. We have several years of pre-treatment data, and 4 years of post-treatment data. In this district, 30 of the 80 schools were identified to participate in the TIF program. In previous work, we have examined whether there is any discernable impact of the Program on student achievement. Now, we turn to the more immediate question of whether teacher behaviors such as retention and transfers can be affected by the TIF program. In particular, we examine whether certain kinds of teachers are more or less affected, whether the size of the bonus matters, and whether teachers of higher or lower value-added appear to respond differently to the TIF program.

Because so many factors together shape the choices of teachers in schools, it is often difficult to determine what causes any observed changes during a given period of time. In the absence of random assignment, the current study determined program-eligibility by a strict formula based on school-level FRPL percentages. This approach created the possibility of conducting two quasi-experimental analyses which compare schools within the district who were and were not assigned to implement the Program:

We are concerned that any observed trends in outcomes within the treatment schools might not be due to the Program, but instead trends may reflect changes in outcomes that were taking place throughout the district. The difference-in-differences (DiD) analysis therefore incorporates the trends in outcomes in a set of matched, non-treated Comparison schools, chosen specifically because they were comparable to the treatment schools in key ways. Because the DiD analysis compares treatment schools to Comparison schools both before and after the implementation of the Program, we can eliminate all unobserved time-invariant factors as potential explanations for the estimated impact of the program. However, the DiD analysis is susceptible to sources of bias from unobserved time-varying factors. We therefore also implement a second methodological approach that has the stronger causal warrant, but has a more limited population of causal inference.

The regression discontinuity (RD) approach relies on comparing the outcomes of schools that were just above and below the threshold for eligibility to participate in the intervention. Because whether a school has 49% or 50% FRPL is arguably random, we expect all factors other than program participation to be very similar.  The RD has a strong causal warrant for identifying the changes in student- and teacher-outcomes that were caused by the Program. The RD analysis has a few short-comings however: The analysis has low statistical power, since there are relatively few schools very “near” the threshold for eligibility. In addition, the RD estimates pertain to schools that have about 50% FRPL students and may not apply to extremely high- or low- SES schools (limited external validity).