Panel: Beyond Prediction: Machine Learning and Causal Inference in Public Policy Research
(Tools of Analysis: Methods, Data, Informatics and Research Design)

Thursday, November 2, 2017: 3:30 PM-5:00 PM
Dusable (Hyatt Regency Chicago)

*Names in bold indicate Presenter

Panel Organizers:  Sarah Tahamont, University of Maryland
Panel Chairs:  Thomas A. Loughran, University of Maryland
Discussants:  Michael Mueller-Smith, University of Michigan and Gregory Stoddard, University of Chicago Crime Lab

Not so Conservative after All: Exact Matching and Attenuation Bias in Randomized Experiments
Sarah Tahamont, University of Maryland, Zubin Jelveh, University of Chicago Crime Lab, Shi Yan, Arizona State University, Aaron Chalfin, University of Pennsylvania and Benjamin Hansen, University of Oregon

Making Case Studies More Credible: Matching, Machine Learning and Model Selection
Valentine Gilbert1, Aaron Chalfin2 and Zubin Jelveh1, (1)University of Chicago Crime Lab, (2)University of Pennsylvania

Beyond Bail: Using Behavioral Science to Improve Timely Court Appearance
Alissa Fishbane, ideas42, Aurelie Ouss, University of Chicago Crime Lab and Anuj Shah, University of Chicago

In recent years, social scientists have introduced methods from computer science to address public policy problems. In particular, machine learning can help address policy challenges that hinge on a prediction (Kleinberg et al, Ludwig et al, Cowgill). But beyond prediction-based questions, machine learning can also help improve estimation methods and thus policy research more generally. This panel consists of four papers that demonstrate how machine learning can help researchers do program evaluation better.

The increasing availability of rich administrative data has helped researchers reduce the cost of data collection for randomized trials. Instead of using original data collection to measure outcomes, researchers can link datasets from government agencies. Two papers on this panel explore how machine learning can improve record linking in administrative data. In the absence of a common unique identifier, researchers use demographic characteristics like name and date of birth to match data sets, but errors in matching are inevitable. In order to minimize errors, researchers often use “exact matching” (same name and date of birth across datasets) in order to ensure that speculative matches do not lead to errors in the dataset that will be used to evaluate the intervention.

The first paper provides evidence that this typical “conservative approach” to linking administrative data in the absence of a common identifier is likely to lead to attenuation bias in the estimation of effects – bias that can be substantial in many instances. The paper characterizes the nature of this bias and demonstrates how machine learning methods for records matching can minimize this problem. The second paper provides a detailed description of how machine learning-based record linking can be done in practice. The approach minimizes false negatives and false positives by optimally learning the rules that would be used by a well-trained researcher.

The remaining two papers leverage machine learning to improve causal estimation. The third paper is concerned with the problem of drawing causal inferences from small-scale case studies, which is extraordinarily common in policy research. The paper presents a new paradigm for implementing synthetic controls methods for case studies, arguing that the quality of a counterfactual unit for a given treatment unit is testable under mild assumptions already taken by implementers of these methods. Using simulated data, the paper shows that the “Super Learner” ensemble prediction algorithm can outperform traditional estimations in which treated and comparison units are closely matched on the basis of pre-treatment trends.

The final paper applies machine learning techniques to estimate treatment heterogeneity within the context of an innovative randomized experiment that tests the efficacy of behavioral science in improving individuals’ compliance with summonses issued by police officers. The paper demonstrates that different nudges can reduce failures to appear in court. But the key question for policymakers is not what works, on average, but what works for whom? Leveraging machine learning the authors characterize the extent to which there is treatment heterogeneity, and how this can inform designing “personalized nudges” to improve policy efficiency.