Association for Public Policy Analysis & Management

Using information extracted from more than 40 rigorous federally-funded evaluations, Abt Associates, in partnership with the Peabody Research Institute of Vanderbilt University, is applying meta-analytic techniques to systematically analyze and summarize the findings generated by the evaluations. The meta-analysis addresses questions about how elements of program design, program implementation, and participant demographics affect program impacts on behavioral outcomes—as well as the extent to which evaluation design and methodology affect the estimation of program impacts. The findings are intended to serve three related purposes: (1) to help program developers design more effective programs; (2) to help practitioners select appropriate programs, allowing them to make a better match between programs, the characteristics of their communities, and their local youth populations; and (3) to help guide funding decisions by federal, state and local entities.

The study had two major components: An Aggregate Data (AD) meta-analysis uses data coded at the study level, which is available for all eligible studies, to examine whether aspects of program design, program implementation, or participant characteristics are associated with program behavioral effects (e.g., are TPP programs more effective if they include a service learning component?), and whether evaluation methods affect the estimation of impacts. This analysis was conducted in a meta-regression framework. An Individual Participant Data (IPD) meta-analysisuses participant-level data obtained from a subset of evaluators to examine whether individual-level participant characteristics such as age, sex, race, and ethnicity are associated with program effects. (E.g., are TPP programs more or less effective for females?). Findings were synthesized using a one-stage approach combining participant-level data from 34 study samples (48,635 youth) with aggregate data from the remaining studies for which participant-level data were not available.

This presentation will present findings from both types of analysis, focusing on a sample of 43 evaluation reports funded by the Office of Adolescent Health and released prior to October 1, 2016. These include 33 final reports and one journal manuscript from grantee-led evaluations, three interim reports from the OAH-led Teen Pregnancy Prevention (TPP) Replication Study, and six interim reports from the OAH-led Pregnancy Prevention Approaches (PPA) Study.