Association for Public Policy Analysis & Management

Investing in evidence-based programs and approaches has caught the attention of some policymakers as a way to target limited resources in ways that will make a difference. The US Department of Health and Human Service (HHS) Office of Adolescent Health’s (OAH) Teen Pregnancy Prevention (TPP) Program is one of six such evidence-based programs across federal agencies launched in 2010 emphasizing the use of evidence and rigorous evaluation.

The TPP Program is a two-tiered program focused on replicating evidence-based programs proven through rigorous evaluation to prevent teen pregnancy (Tier 1) and developing and testing additional models and innovative strategies for preventing teen pregnancy (Tier 2). The TPP evidence base established in 2010, in preparation for the TPP Program, was the first time a systematic review was conducted on TPP. All but one of the programs identified demonstrated evidence of effectiveness with a single study, often conducted by the program developer. OAH’s goals with the TPP Program evaluations were to build a body of evidence about where, when, and with whom individual evidence-based programs are effective, and to contribute new programs having some evidence of effectiveness to be replicated and further evaluated in the future.

OAH funded 41 TPP evaluations, through both grantees and contractors. The Tier 1 evaluations implemented evidence-based programs as intended and with quality (replications), but in new settings and with new populations. The Tier 2 evaluations were intended to identify new TPP programs, or significant adaptations of evidence-based programs, effective in reducing teen pregnancy. Nineteen replication evaluations were conducted on 10 different TPP program models. Twenty-two evaluations were conducted on 22 new TPP program models. All of OAH’s TPP evaluations adhered to the rigorous research standards of the HHS Evidence Review, collected baseline and follow-up data, at least one behavioral outcome measure from the HHS Evidence Review, and reported on a comprehensive set of performance measures every six months, to ensure consistency of measures for comparability and for later synthesis. Evaluation report templates were used to ensure reporting consistency. Utilizing the impact reports and performance measure data, OAH conducted a synthesis of the TPP Program evaluations. The evaluations were coded into three categories – replicated/positive impacts; did not replicate/no positive impacts; or inconclusive. The replication impact findings were mixed; the Tier 2 evaluations yielded several new effective TPP programs.

This presentation will discuss the effort to move the evidence-base in TPP beyond single, developer-conducted studies to a broader, deeper body of evidence that consumers can use to choose the best program for their needs; how lessons learned were used to redesign the TPP program; and current efforts to make sense of mixed replication evaluation findings.