Association for Public Policy Analysis & Management

While rigorous impact evaluations are often national in scope, most education policy is local, at the school or district level. A key question for evidence-based policy is whether the evidence from national studies help local decision-makers make better decisions.

This paper addresses the question of how well local decisions in education are informed by published findings from randomized control trials (RCTs). First, it explains why reported evidence from RCTs may not accurately predict the impacts of adopting an intervention in individual localities: Local predictions based on evidence from other localities will contain both bias and sampling error. Second, it offers a set of methods for simulating the use of results from a multi-site RCT in sites outside the evaluation sample, quantifying the accuracy of the local predictions that can be obtained from multi-site RCTs, and assessing the likelihood that prediction errors will lead to errors in local policy decisions. Third, it provides the first empirical evidence on the accuracy with which local impacts can be predicted from published evidence from RCTs and on the risk of making the local policy errors based on this evidence. The paper finds that predictions of local impacts from national studies can contain substantial errors that can lead to a high risk of making the wrong policy decision. These results should lead the field to consider how best to generate evidence that will provide better guidance to local policymakers.