Association for Public Policy Analysis & Management

Successful policy and program implementation is about creating change in the target population, as well as change in the implementation system. Both types of change have been discussed in varying degrees in prior literature, and are often the focus of evidence based evaluations (Weiss et al. 2014; Hargreaves, 2010).  While policy analysts and program evaluators are most often interested in whether or not an intervention improves the condition or behavior of the target population, sustainable interventions often require changes to the implementation system (Sandfort and Moulton 2015). A growing body of literature in implementation science focuses on the extent to which interventions become successfully integrated within implementation systems (Fixsen et al. 2005).    

In addition to changes in the system and target population, there is a third type of change that is often overlooked or even discouraged in evidence based evaluations: change in the program technology to be implemented. Such change runs counter to the notion of “fidelity” to the program design, often considered a critical component of rigorous evaluations. Fidelity to the program design depends upon a precisely delineated program logic, a clearly specified implementation plan, and well-defined outcomes (Weiss et al. 2014). While perhaps beneficial for evaluating causality in a controlled field experiment, such precision does not reflect reality and may not be desired.  Fidelity may impede system change, target population change, and ongoing improvement so often necessary in tackling complex policy problems.  We purport that implementation is more successful when the program technology is adapted to fit the constraints of the system and needs of the target population.

In this manuscript, we present an evaluation framework that integrates these three types of change as occurring simultaneously in a dynamic system. Building from our prior scholarship, we conceive of the implementation system as nested multi-level strategic action fields, bounded by sources of legitimate authority at each level that shape the core program technology (Sandfort and Moulton 2014; Moulton and Sandfort 2015). Rather than an emphasis on “fidelity” to program design, policy and program evaluators can play an important role in facilitating and evaluating changes to the system, target population and program “fit.”  Such a framework can help inform future research agendas that bridge policy analysis, evaluation and management, and can help facilitate improved implementation practice within complex systems. 

We demonstrate the usefulness of this framework by applying it to two different rigorous evaluations of policy and program innovations taking place in municipal governments: an evaluation of an innovation to link delinquent water utility customers to financial empowerment services, and an evaluation of how changes in state welfare performance measurement influences system operations and outcomes.  Rather than simply evaluating whether or not the innovation was successful, we use the framework to identify the complex interactions that facilitate (and impede) system change, changes in program technology and ultimately change in the target populations.  Making sense of how policies and programs are carried out to address messy public problems requires researchers to embrace, rather than reduce, the complexity found in implementation systems.