Association for Public Policy Analysis & Management

Panel: Learning about and from Variation in Program Impacts Using Multisite Trials

(Tools of Analysis: Methods, Data, Informatics and Research Design)

The recent increase in the number of large-scale multi-site random assignment evaluations has generated increased interest in understanding variation in program effectiveness. The information that can be obtained from such analyses about for whom and in what contexts a program works best/worst has important implications for: (a) targeting of resources (e.g., can we provide services to those who are likely to benefit from them the most?), (b) increasing equity (e.g., does the program bring the “bottom up” and reduce inequity?), and (c) improving programs (e.g., how can we modify the program to better serve its target population?). Moreover, natural variation in treatment contrasts, treatment effects on mediators, and treatment effects on outcomes may provide opportunities to learn lessons about the mechanisms through which programs work (when they do so). This panel will explore innovative analytic approaches for learning more about and from variation in program impacts. After introductory remarks, the four paper presentations will discuss innovative strategies, challenges to the application of these strategies and/or one or more examples of substantive insights gained from investigating variation in programmatic impact. The first presentation will focus on strategies for assessing the presence of cross-site variation and will present estimates of the magnitude of this variation in the context of 10 large, multi-site education-focused trials. The second presentation will focus on a multiple-site, multiple mediator instrumental variables (MSMM-IV) strategy to address the case in which one or more potential mediators are unobserved and the standard IV exclusion restriction assumption is therefore invalid. The third presentation will discuss an innovative weighting strategy for statistical inference of the population average and the between-site variation of mediated effects. Finally, the fourth presentation will discuss the challenges associated with model-based estimation of principal causal effects and the promise of bound-based estimation. The panel will include substantial discussion about the methods and related findings, including their implications for study planning, generalization, and theories regarding when effects vary most/least across sites.