Association for Public Policy Analysis & Management

To evaluate the effects of federal programs and a wide range of social interventions, policymakers have increasingly turned to Randomized Controlled Trials (RCTs).  To obtain an adequate sample—and perhaps to bolster the generalizability of study findings—many RCTs are conducted in multiple sites.  When the impacts of an intervention vary across sites, the particular sites selected for a multi-site RCT may heavily influence the findings. While most RCTs select sites nonrandomly (see Olsen et al., 2013), random site selection may warrant additional usage to obtain representative samples and evidence that generalizes beyond the study sample to populations of interest to policymakers. 

This paper provides plausible estimates of the sample size requirements for multi-site RCTs in education that select schools randomly.  These estimates are based heavily on already published evidence from large-scale randomized trials on the variation in impacts across sites—a key statistical power parameter for this type of design.  In addition, we contribute one additional piece of evidence on cross-site impact variation from a study of charter schools, which was selected because the effects of charter schools almost surely vary more than the effects of most educational interventions.  This allows us to put a plausible upper bound (though clearly not a mathematical upper bound) on the cross-site impact variance and thus the sample size requirements for multi-site RCTs in education that select sites randomly.