Saturday, November 10, 2012: 10:15 AM-11:45 AM
Washington (Sheraton Baltimore City Center Hotel)
*Names in bold indicate Presenter
Organizers: Thomas Cook, Northwestern University IPR
Moderators: Frank Yoon, Mathematica Policy Research, Inc.
Chairs: Jennifer Steele, RAND Corporation
There are circumstances where randomized controlled trials (RCTs) are not possible and observational studies are required. Theory clearly articulates the assumptions that are required for an observational study to produce unbiased causal estimates. However, it is difficult in practice to determine whether these assumptions have been met. Recourse is needed to empirical within study comparisons (WSCs) of experimental and quasi-experimental approaches. In WSCs, the researcher compares causal effects from an experiment with those from a quasi-experiment when each has the same treatment group or a theoretically equivalent one. What varies, therefore, is how the comparison group is formed - at random in the experiment or systematically in the observational study. Should a properly conducted experiment and a statistically adjusted observational study result in similar estimates over a number of such WSCs, this conclusion strengthens our practical knowledge about the conditions under which the type of observational study under analysis produces unbiased estimates. This should then provide some guidance as to the quasi-experimental design and analysis options that warrant most use because they often reproduce experimental estimates.
Introduced by LaLonde (1986), the earliest within-study comparison designs used data from job training evaluations and compared results from an observational study with those from an experimental benchmark. More recent implementations of WSCs have examined the performance of these approaches in the fields of education, medicine, and early childhood development, as well as looked at how various design features (such as the pretest, or reliability of the covariates for matching) performed in producing unbiased treatment effects. Theory on the proper design, implementation, and analysis of within-study comparisons has also emerged.
The papers in this session reflect this evolution. Specifically, they probe the role of so-called true and proxy pretests at one or two pre-intervention time points, the role of the conceptual heterogeneity and number of the covariates used, and the performance of sequential matching procedures when researchers draw comparisons first from schools and then from students within schools. All the results show strong levels of correspondence between the experimental and matched observational study results.