As statistical theory on non-experimental methods continues to develop, more WSCs will be needed to assess whether these methods are suitable for causal inference in field settings, that is, whether the underlying stringent assumptions required for identification and estimation are likely to be met. For example, empirical validation studies are needed to examine the performance of observational approaches (such as multilevel matching methods and prognosis scores), and non-experimental methods that utilize repeated measures over time, such as pretest-posttest, difference-in-differences, and the interrupted time series designs. WSCs also provide opportunities for researchers to check their hypotheses about statistical models for addressing selection bias. At stake is a methodology that allows researchers to develop and refine empirically based “best practices” for non-experimental approaches.

Despite the opportunities and reasons for conducting WSCs, the approach is underutilized as a tool for improving research practice in evaluation settings. This is because there is no coherent theory of WSC designs as a method for conducting research on non-experimental approaches. The lack of theory guiding the design, implementation, and analysis of WSCs is problematic for a number of reasons. First, for researchers who wish to use WSCs to investigate non-experimental methods, the only available resources are examples of WSC designs scattered across the literature on job training, criminology, education, political science, international development, public health, and education. With the exception of a brief discussion by Cook, Shadish, and Wong (2008) who present six criteria for a causally valid WSC, there is no methodological paper devoted to the appropriate design and analysis of the WSC itself. As a result, the existing WSCs are of heterogeneous quality, with researchers using ad hoc designs and methods that may or may not be appropriate for addressing the research question of interest (Shadish, Steiner & Cook, 2012). Second, without a general theory of WSCs, researchers do not have a framework for understanding different types of WSC approaches, nor for comparing the approaches’ relative strengths and limitations for addressing the research questions at hand. Finally, thus far, the WSC design has been used by a relatively small cadre of research methodologists (and scholars interested in methods) who wish to understand the performance of non-experimental methods in field settings.

This paper presents a coherent theory of the design, implementation and analysis of WSCs for evaluating non-experimental methods. It introduces and identifies the multiple purposes of WSCs, required design components, common threats to validity, design variants, and causal estimands of interest in WSCs. It highlights the need for a WSC research protocol to avoid inadvertent skewing of results, and addresses important components of a WSC research protocol. Finally, the paper addresses methodological approaches for establishing correspondence experimental and non-experimental results in a WSC design.