Indiana University SPEA Edward J. Bloustein School of Planning and Public Policy University of Pennsylvania AIR American University

Panel: Sample Size Requirements for RCTs with Varying Impacts
(Tools of Analysis: Methods, Data, Informatics and Research Design)

Thursday, November 12, 2015: 8:30 AM-10:00 AM
Pearson I (Hyatt Regency Miami)

*Names in bold indicate Presenter

Panel Organizers:  Robert Olsen, Rob Olsen LLC
Panel Chairs:  John Deke, Mathematica Policy Research
Discussants:  Winston Lin, Abt Associates

Analysis of Statistical Precision for Multi-Site Randomized Trials
Howard Bloom, MDRC and Jessaca Spybrook, Western Michigan University

Sample Size Requirements for Education Multi-Site RCTs That Select Sites Randomly
Robert Olsen1, Edward Bein2 and David Judkins2, (1)Rob Olsen LLC, (2)Abt Associates

Impact variation is often the focus of analysis in impact evaluations. Policymakers may want to know whether some groups benefit more than others so that the program can be targeted more efficiently—and to identify populations that are poorly served by existing programs. In other cases, impact variation is a “nuisance” that impedes researchers in estimating population average treatment effects. Rigorous impact studies are rarely conducted in a random sample of sites from the population of interest; these studies can lead to biased estimates of population average treatment effects if the impacts of the intervention vary. This panel is focused on the sample size requirements for impact evaluations when impacts vary. The first paper in this panel provides a comprehensive assessment of statistical power for multi-site randomized trials for estimating average effects and detecting variation in impacts. The remaining papers focus on impact studies in education and estimate the sample size requirements for studies that estimate population average treatment effects when impacts vary. The second paper uses empirical evidence to estimate sample size requirements when schools are selected randomly to be representative of the population of policy interest; the third paper estimates sample size requirements when schools are selected systematically to match the observed characteristics of the population.