Association for Public Policy Analysis & Management

This paper describes the use of evaluation designs that address the challenge of generating rigorous evidence about the effectiveness of place-based initiatives aimed at changing population-level outcomes. Place-based initiatives that focus on population-level changes in health and well-being outcomes have been the focus of many federal and philanthropic investments in recent years. However, few have produced evaluations that can confidently attribute community change to the initiative. Synthetic control and comparative interrupted time series methods are designed to exploit “spillovers” where the experimental design assumptions are not met.

In this paper, we apply these methods to the challenge of designing an evaluation of the Office of Adolescent Health’s (OAH) Teen Pregnancy Prevention strategy focused on community-wide initiatives. OAH funded 50 organizations across the United States in 2015 with five-year grants to (1) implement evidence-based programming to scale in communities with the greatest need, (2) systematically mobilize, engage, and build support in communities, and (3) connect adolescents to youth-friendly health services through coordinated referral systems. Targeted communities are defined by a variety of geographic boundaries such as zip codes, school districts, cities, and counties, and include highly urban as well as highly rural areas.

These initiatives are intended to saturate specific geographic footprints with a focus on changing social norms related to sexual risk-taking behavior and decreasing the teen birth rate, but the methods can be applied to other place-based interventions focusing on, for example, education, poverty, housing, or criminal justice outcomes. The presentation will use geographic and historical outcome data to show how we are approaching the use of synthetic control and comparative interrupted time series methods to construct appropriate comparison groups that will allow for a rigorous quasi-experimental impact analysis. Challenges encountered with the design will be highlighted, as well as how we have addressed them.