How Much Can External Validity Bias Be Reduced by Aligning Sample and Population on School District Characteristics?
*Names in bold indicate Presenter
This paper examines the extent to which the combination of standard statistical methods and publicly available data on school district characteristics can reduce the external validity bias from conducting impact studies in purposive samples. The paper applies standard regression and matching methods to assess whether adjusting for district size, urbanicity and economic disadvantage, as measured in Stuart et al., substantially reduces the bias from purposive site selection, as estimated in Bell et al. If not, impact evaluations may need to either collect more nuanced data on treatment effect moderators to reduce the bias or select sites randomly to obtain a more representative sample