Association for Public Policy Analysis & Management

Panel: Advances in Experimental and Quasi-Experimental Design

(Tools of Analysis: Methods, Data, Informatics and Research Design)

As the policy research community grows ever more sophisticated in the questions that it asks about programmatic impact, so too must our analytic methods grow to meet that sophistication. This panel will include four papers covering advances in experimental or quasi-experimental design and data analysis that are motivated by real-world policy applications. Given the prevalence of intervention implementation outside of the framework of a randomized trial, matching strategies are an important tool for policy analysis. Yet, how should matching be generalized and conducted for non-binary treatments? The first paper in our panel addresses this question through the development of a sequential matching strategy using a generalized version of the propensity score. If a treatment is implemented at the site-level (for example, if a treatment is applied to an entire school), how should matching for the estimation of treatment effects be conducted in a multi-level framework? The second paper in our panel will take up this question, and will compare experimentally derived treatment effects to those obtained through an optimal multi-level matching strategy. Turning next to multi-site experimental trials, what strategy should we use to estimate the distribution of site-level treatment effects? A default and convenient assumption is normality of site-level effects, but is this assumption justified? If not, what are the implications for our substantive conclusions about site-level impact variation? The third paper in our panel will take up these questions. If we seek to generalize impact estimates to a broader population of interest, how can we do so? Specifically, how well can regression or reweighting methods work to translate internally valid impact estimates to externally valid population average treatment effects? The final paper in our panel will address these questions, shedding light on when selected methods for generalization do and do not work well. The panel will be accessible to applied data analysts and will include a discussion of implications and recommendations for practice.