Association for Public Policy Analysis & Management

Policymakers and others are increasingly interested in replicating and scaling up effective educational programs. Frequently, however, they encounter a critical road block in doing so: insufficient information about the resources and corresponding costs associated with implementing a particular policy or program. While it is critical that we understand which policies work and for whom, summative evidence of a policy’s or program’s effects falls short of providing decision makers with information about how the program was implemented for success. This is especially true when it comes to information about the package of tangible ingredients that were put in place to attain observed effects. Itemizing these ingredients is not only important for understanding what might be required for program implementation, they also are the basis for evaluating the feasibility – both in terms of available resources and potential costs – undertaking or expanding a particular policy or program.

In this paper we offer three rationale, with empirical applications, for integrating resource and cost information into evaluations of program effects. Specifically, we consider three applications for decision making:

1) Developing a resource “recipes” that informs future efforts to implement a policy or program.

We show how a rich understanding or program ingredients supports efforts to replicate effective educational program and bring them to scale. In particular, we illustrate how characterizing both the quantity and qualities of the resources used by effective programs provide importance guidance to others looking to adopt or expand policies and programs elsewhere.

2) Understanding variation in implementation across sites and contexts in ways that unpack different models decision makers might adopt when replicating and scaling-up an effective program.

Decades of research examining education policy and program implementation points to the important role played by context and capacity in shaping local responses to education reforms and, more recently, particularly at the federal and state levels, education policies increasingly embed assumptions that allow for, and even encourage, local variation in program implementation. Identifying “ingredients” that go into producing observed effects shed light on how different resource packages might explain differences in program impacts across sites, as well as highlight different strategies for deploying resources to attain stated goals.

3) Estimating program costs to support decision making about cost feasibility.

At the most basic level, decision makers need to understand whether they can afford its implementation and ongoing operations. Estimating cost to determine feasibility is an essential consideration for program replication and expansion.

In each application, the paper offers empirical, case based examples of how the “ingredients method” (Levin & McEwan, 2001) was used to understand program implementation in ways that support replicating and scaling up policies and programs that work. In doing so, we demonstrate how coupling data on resources and costs with estimates of program impacts can support decision making.