Panel: Practical Matching Conditions In Observational Studies That Often Reproduce RCT Estimates
(Methods)

Saturday, November 10, 2012: 10:15 AM-11:45 AM
Washington (Sheraton Baltimore City Center Hotel)

*Names in bold indicate Presenter

Organizers:  Thomas Cook, Northwestern University IPR
Moderators:  Frank Yoon, Mathematica Policy Research, Inc.
Chairs:  Jennifer Steele, RAND Corporation

There are circumstances where randomized controlled trials (RCTs) are not possible and observational studies are required. Theory clearly articulates the assumptions that are required for an observational study to produce unbiased causal estimates. However, it is difficult in practice to determine whether these assumptions have been met. Recourse is needed to empirical within study comparisons (WSCs) of experimental and quasi-experimental approaches. In WSCs, the researcher compares causal effects from an experiment with those from a quasi-experiment when each has the same treatment group or a theoretically equivalent one. What varies, therefore, is how the comparison group is formed - at random in the experiment or systematically in the observational study. Should a properly conducted experiment and a statistically adjusted observational study result in similar estimates over a number of such WSCs, this conclusion strengthens our practical knowledge about the conditions under which the type of observational study under analysis produces unbiased estimates. This should then provide some guidance as to the quasi-experimental design and analysis options that warrant most use because they often reproduce experimental estimates. Introduced by LaLonde (1986), the earliest within-study comparison designs used data from job training evaluations and compared results from an observational study with those from an experimental benchmark. More recent implementations of WSCs have examined the performance of these approaches in the fields of education, medicine, and early childhood development, as well as looked at how various design features (such as the pretest, or reliability of the covariates for matching) performed in producing unbiased treatment effects. Theory on the proper design, implementation, and analysis of within-study comparisons has also emerged. The papers in this session reflect this evolution. Specifically, they probe the role of so-called true and proxy pretests at one or two pre-intervention time points, the role of the conceptual heterogeneity and number of the covariates used, and the performance of sequential matching procedures when researchers draw comparisons first from schools and then from students within schools. All the results show strong levels of correspondence between the experimental and matched observational study results.

Prospectively Choosing Comparison Units Using Sequential Matching Procedures
Vivian Wong, University of Virginia, Kelly Hallberg, Northwestern University, Peter Steiner, University of Wisconsin - Madison, Thomas Cook, Northwestern University IPR and Nathan Jones, Educational Testing Service



Covariate Selection for PS Designs In the Absence of Substantive Theories
Peter Steiner, University of Wisconsin - Madison, Thomas Cook, Northwestern University IPR and Wei Li, Northwestern University



Testing Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates for Charter Schools
Kenneth Fortson, Natalya Verbitsky-Savitz, Emma Kopa and Phil Gleason, Mathematica Policy Research




See more of: Methods
See more of: Panel