Association for Public Policy Analysis & Management

We compare the treatment effect estimate from a randomized controlled-trial (RCT) design to that of a regression discontinuity (RD) design, both at the assignment variable cutoff and away from the cutoff. The RCT data come from a get-out-the-vote field experiment conducted during the 2002 Michigan gubernatorial election (Nickerson et al. 2006). The experiment randomized households across 14 state house districts to receive volunteer phone calls emphasizing the importance of voting. Taking advantage of the targeted age group of the RCT (registered voters under the age of 35), we first create a traditional RD using the treatment group from the RCT, discarding the control observations, and instead use all registered voters above the age of 35 in the same house districts as the RD comparison group. The discontinuity in treatment status at age 35 serves as the identification for the RD design. The shared treatment group in the RCT and RD eliminates many potential confounding effects, particularly the content of the intervention and its implementation; this design comparison is called a within-study comparison (WSC). Building on the findings of previous WSC research and the guidelines set out in Chiang et al (forthcoming) and Cook et al. (2008), we re-estimated the RCT, limiting to voters within 5 years of the cutoff age, thereby obtaining a local area treatment effect similar to the estimand obtained from an RD analysis. We verified balance in this restricted sample and used a linear model conditional on key covariates to obtain the benchmark estimate. We used a local linear regression with optimal bandwidth (Imbens & Kalyanaraman 2012) and the same key covariates as in the RCT model to estimate the RD effect. The resulting RCT and RD estimates are highly correspondent in direction, magnitude, and significance. We used a bootstrapped variance estimate of the difference in the design impacts to construct a 95% confidence interval around the correspondence estimate.

Building on the experimental replication at the cutoff, we then construct a comparative regression-discontinuity (CRD) by adding back in the experimental control observations, thus creating a CRD-comparison group design (Tang et al. forthcoming). This allows us to estimate the counterfactual outcome away from the discontinuity cutoff but within the treated range of the assignment variable. This creates the potential of unbiased RD estimates anywhere along the treated portion of the assignment variable, assuming parallel outcome regression lines. Following Tang et al. (forthcoming) and Chiang et al. (forthcoming) we estimate the CRD away from the cutoff and compare with similar experimental estimates. The variance of the RCT-CRD contrasts was again calculated with bootstrapping. We also compare CRD estimates with RCT estimates. We conclude that the regression-discontinuity design can replicate experimental findings at the cutoff, with impact estimates well within tolerable bounds. We also provide more evidence that under certain conditions comparative regression-discontinuity is a promising method for extrapolating RD estimates away from the cutoff, thereby potentially extending the applicability of RD designs.