Association for Public Policy Analysis & Management

Panel: Sample Size Requirements for RCTs with Varying Impacts

(Tools of Analysis: Methods, Data, Informatics and Research Design)

Impact variation is often the focus of analysis in impact evaluations. Policymakers may want to know whether some groups benefit more than others so that the program can be targeted more efficiently—and to identify populations that are poorly served by existing programs. In other cases, impact variation is a “nuisance” that impedes researchers in estimating population average treatment effects. Rigorous impact studies are rarely conducted in a random sample of sites from the population of interest; these studies can lead to biased estimates of population average treatment effects if the impacts of the intervention vary. This panel is focused on the sample size requirements for impact evaluations when impacts vary. The first paper in this panel provides a comprehensive assessment of statistical power for multi-site randomized trials for estimating average effects and detecting variation in impacts. The remaining papers focus on impact studies in education and estimate the sample size requirements for studies that estimate population average treatment effects when impacts vary. The second paper uses empirical evidence to estimate sample size requirements when schools are selected randomly to be representative of the population of policy interest; the third paper estimates sample size requirements when schools are selected systematically to match the observed characteristics of the population.