Indiana University SPEA Edward J. Bloustein School of Planning and Public Policy University of Pennsylvania AIR American University

Panel: Innovations in Evaluation for Improved Internal and External Validity
(Tools of Analysis: Methods, Data, Informatics and Research Design)

Saturday, November 14, 2015: 1:45 PM-3:15 PM
Foster I (Hyatt Regency Miami)

*Names in bold indicate Presenter

Panel Organizers:  Robert Olsen, Rob Olsen LLC
Panel Chairs:  Robert Olsen, Rob Olsen LLC
Discussants:  Jeffrey Smith, University of Michigan and Thomas Cook, Northwestern University

Propensity Scores and Causal Inference Using Machine Learning Methods
Austin Nichols, Abt Associates and Linden McBride, Cornell University

External Validity in Fuzzy Regression Discontinuity Designs
Marinho Bertanha, Center for Operations Research and Econometrics - Université catholique de Louvain

The evaluation field has historically focused on isolating the intervention effects from other potentially confounding factors to ensure that the evaluation results have high internal validity. Random assignment yields evaluations with high internal validity when attrition is low. However, the designs with the strongest internal validity sometimes have limited external validity. Randomized trials conducted in a convenience sample of sites may produce results that do not generalize to the broader population of sites. Regression discontinuity designs produce estimates for values close to the cutoff—but the results may not generalize to other points in the distribution. This panel includes papers designed to advance the internal and external validity of impact evaluations. The first paper is focused on improving the internal validity of propensity score matching estimators through machine learning. The second paper is focused on improving the external validity of Randomized Controlled Trials when sites are selected purposively. The last two papers are focused on assessing and enhancing the external validity of Regression Discontinuity Designs for points away from the cutoff—using methods that require high internal validity.