Association for Public Policy Analysis & Management

Randomized Controlled Trials (RCTs) are the “gold-standard” for estimating causal impacts of educational programs. Students subject to lotteries, however, often are not representative of the broader population of students experiencing the educational treatment. With few exceptions, researchers are not able to determine how much selection bias exists when various quasi-experimental approaches are used in place of experimental ones within a school choice context. We are left wondering about the magnitude of the internal-for-external validity tradeoff that education researchers often face.

This study assesses the extent to which methods such as propensity score matching or observational models with control variables can replicate the “benchmark” experimental results of the District of Columbia Opportunity Scholarship (DC OSP) school voucher evaluation. The federal private school voucher program is an exemplar subject for study because self-selection is assumed to be a major influence on whether or not a low-income urban student attends a private school. We treat Instrumental Variables Analysis (IV) estimates of the impact of private schooling on student outcomes, some of which are being presented for the first time in this study, as the causal “benchmark” estimate. While our data are fairly limited, and the results relatively imprecise, we find preliminary evidence that covariate choice matters, and that method choice matters, but perhaps only when comparing to a broader sample that includes students who did not apply to the program.

Interestingly, we find that the direction of the estimation bias that we detect from some of the quasi-experimental approaches is positive when the sample is limited to program applicants, but negative when it is expanded to include non-applicants. This finding suggests that the applicants to means-tested school voucher programs are negatively selected, but the subgroup of applicants who actually use a voucher if offered tend to be positively selected.