Panel: Using Systematic Reviews for Evidence-Based Decision Making In Policy and Practice

Thursday, November 8, 2012: 3:00 PM-4:30 PM
McKeldon (Sheraton Baltimore City Center Hotel)

*Names in bold indicate Presenter

Organizers:  Sarah A Avellar, Mathematica Policy Research
Moderators:  Lauren Supplee, Administration for Children and Families
Chairs:  Jennifer Romich, University of Washington

As policymakers and government leaders consider whether and how to invest scarce resources in services and programs for children, families, and those in need, evidence-based decision making continues to grow as the current “coin of the practice and policy realm” (McCall, 2009). In his official blog, former OMB Director Peter Orszag emphasized that decisions should be based on “strong evidence that carefully targeted investments will produce results” (Orszag, 2009). Evidence reviews may be an efficient way to examine program effectiveness as they draw on the lessons and conclusions from existing research. In an evidence review, research is assessed with a set of standards to evaluate its quality on components such as study design and execution, measurement quality, and appropriateness of the analysis. If more than one study of a program meets the review’s standards, results may be summarized or compiled to provide a complete picture of the research evidence base. Currently, the federal government oversees reviews on topics, including but not limited to education; family strengthening; home visiting; mental health and substance use; medical screening, diagnostics, and therapeutic intervention; and teen pregnancy prevention. Evidence reviews, however, are not without challenges since the research may use disparate methods and have conflicting conclusions. Further, the review may be hampered by limited, out-of-date, or otherwise inaccessible information. Therefore, researchers must develop methods that can be applied consistently across studies for dealing with issues, such as missing information or inconsistent results, and answering what constitutes “evidence.” As incontrovertible standards often do not exist, these strategies may differ, which may be confusing to the typical user of the information. This panel will bring together researchers who are leading systematic reviews of evidence in three different areas, and federal agency staff who have used the systematic review results for decision making. The evidence reviews include: • Home Visiting Evidence of Effectiveness review assesses the research on home visiting program models that serve families with pregnant women and children from birth to age 5. • National Registry of Evidence-Based Programs and Practices rates the strength of the evidence supporting the outcomes of mental health promotion, substance abuse prevention, and mental health and substance abuse treatment interventions as well as the amount and general quality of the resources available to support the use of these interventions. • HHS Teen Pregnancy Prevention Evidence Review is a comprehensive review of teen pregnancy prevention programs and initiatives. To inform policy decisions, each of these reviews has developed a rigorous, thorough, and transparent process for assessing the research and evaluation literature. During the panel, participants will discuss how the systematic reviews were conducted and lessons learned, highlighting similarities and differences across the topic areas, and discussing the strengths and limitations of their respective bodies of research. The discussant will focus on the benefits and challenges of using systematic review of evidence for decision making.

The National Registry of Evidence-Based Programs and Practices
Stephen Gardner, MANILA Consulting Group, Inc.

Findings From the HHS Teen Pregnancy Prevention Evidence Review
Brian Goesling, Mathematica Policy Research

See more of: Methods
See more of: Panel