Association for Public Policy Analysis & Management

Purposive site selection in rigorous impact evaluations can lead to biased estimates for the population from which the sample was selected if the selection process favors sites with larger or smaller than average impacts (Olsen et al., 2013). Recent research provides evidence that the school districts which participate in rigorous impact studies are larger, more urban, and more disadvantaged than the average school district (Stuart et al., 2017). Related research provides evidence that a hypothetical evaluation of a federal reading program conducted in districts that participated in prior randomized trials would lead to biased impact estimates for the broader population (Bell et al., 2015).

This paper examines the extent to which the combination of standard statistical methods and publicly available data on school district characteristics can reduce the external validity bias from conducting impact studies in purposive samples. The paper applies standard regression and matching methods to assess whether adjusting for district size, urbanicity and economic disadvantage, as measured in Stuart et al., substantially reduces the bias from purposive site selection, as estimated in Bell et al. If not, impact evaluations may need to either collect more nuanced data on treatment effect moderators to reduce the bias or select sites randomly to obtain a more representative sample