Association for Public Policy Analysis & Management

This paper presents a meta-analysis (MA) of 231 contrasts each of which compares an impact estimate obtained by Regression discontinuity (RD) with an impact estimate from a similar randomized control trial (RCT). These contrasts come from 14 within study comparison (WSC) studies that were each designed to test how biased RDs are compared to RCTs. The WSCs are each characterized by RCTs and RDs that have the same treatment group and by most of them estimating the causal effect at the cutoff value determining treatment assignment in RD where theory predicts that causal estimates should be identical in expectation. But for this to occur in individual studies requires that the RCTs and RDs should be perfectly implemented. Since neither can be we account for RD and RCT quality in our analyses. The present study meta-analyses a quite diverse set of contrasts that vary both within and across WSCs and enables us to test how closely the average RCT and RD estimates correspond. In general little evidence of bias is found even when looking at studies with average RD and RCT quality.

 The methodology used blends two research traditions. The first is the within study comparison (WSC) literature that seeks to establish whether RCTs and quasi-experiments (QEs) produce equivalent or different causal estimates when each design shares the same treatment group. Such studies, also called design experiments, use the RCT as the unbiased causal benchmark and assess the degree of final bias by comparing the RCT estimate to the adjusted posttest difference in the QE. What varies in a WSC, therefore, is how the comparison group is formed – at random in the RCT and systematically in the QE. In the RD case, the systematic assignment procedure is via a cutoff allocation mechanism that, in the sharp case, completely determines treatment allocation.  When the two final impact estimates are identical, no bias is evident; when they differ, bias is evident and the type of QE design used and the type of adjustments used to control for selection bias are held to be inadequate.

 Our analysis accounts for clustering of contrasts within WSCs and we use both frequentist and Bayesian methods to test the robustness of our results and to facilitate interpretation for a diverse audience.