Using Rigorous Evaluation Results to Improve Local Policy Decisions
*Names in bold indicate Presenter
This paper addresses the question of how well local decisions in education are informed by published findings from randomized control trials (RCTs). First, it explains why reported evidence from RCTs may not accurately predict the impacts of adopting an intervention in individual localities: Local predictions based on evidence from other localities will contain both bias and sampling error. Second, it offers a set of methods for simulating the use of results from a multi-site RCT in sites outside the evaluation sample, quantifying the accuracy of the local predictions that can be obtained from multi-site RCTs, and assessing the likelihood that prediction errors will lead to errors in local policy decisions. Third, it provides the first empirical evidence on the accuracy with which local impacts can be predicted from published evidence from RCTs and on the risk of making the local policy errors based on this evidence. The paper finds that predictions of local impacts from national studies can contain substantial errors that can lead to a high risk of making the wrong policy decision. These results should lead the field to consider how best to generate evidence that will provide better guidance to local policymakers.