Association for Public Policy Analysis & Management

Research clearinghouses have become more and more prominent in the recent times, in line with the increasing emphasis on evidence-based public policy. This paper reviews the methodological standards of evidence that clearinghouses use for assessing the quality of causal research studies and also for assigning them one of several ratings of overall program effectiveness. The variation in causal method standards is then used to understand how consistent the clearinghouses are when they rate the same program.

The landscape of research clearinghouses is characterized by immense diversity: clearinghouses differ greatly in policy domains, scope, functions, financial sources and institutional structure. A primary activity carried out by research clearinghouses is to rate policies, programs and/or practices on the basis of their effectiveness as demonstrated by research evidence. This paper examines and analyzes the criteria used by clearinghouses to adjudge the effectiveness of programs. The rating criteria employed by clearinghouses cover a wide range of factors such as research quality, sampling, data collection, implementation factors, program outcomes and dissemination/replication potential. Our analysis shows that research design carries crucial weight in the quality assessment of research studies, which is one of the most important factors determining the overall effectiveness rating of programs. Given the centrality of research design, this paper pays specific attention to the methodological standards of evidence that apply to each of the following commonly known research designs namely: randomized control trials (RCT), regression discontinuity designs (RDD), interrupted time series (ITS), instrumental variables (IV) and other quasi-experimental designs. We see that while there is considerable uniformity across the standards of evidence for RCTs, there is tremendous variation in how clearinghouses value and assess studies with quasi-experimental designs.

Thereafter, we exemplify how the same program is rated by different clearinghouses and how the differences in the methodological standards of evidence for research designs may bring about differences in program effectiveness ratings. Six case studies (i.e. six programs) are examined where clearinghouse differences emerge in how the same program is rated. Finally, this paper discusses the implications of such differences in methodological standards of evidence that might result in different effectiveness ratings of programs. As there are a significant number of clearinghouses operating in the United States, it is important to understand how program effectiveness is assessed across the board so that we can understand how to better use research evidence to inform public policy.