Association for Public Policy Analysis & Management

Pay for Success (PFS) is an innovative financing model where the private investor provides upfront capital to fund effective preventative interventions that are expected to generate cost savings for government. Some recent examples of interventions that may save the public sector more money than they cost include recidivism prevention programs, supportive housing and employment programs, and prenatal and early childhood interventions. The government involved in a PFS transaction uses the cost savings to repay the investors with interest, contingent on the program performance on pre-defined metrics measured through a rigorous evaluation. The role of impact evaluation, thus, becomes critical in this model, as the decision on whether and how much to repay the private investor is hinged upon the outcome of a careful evaluation by an impartial evaluator.

For the PFS project to be successful, the benefits to the government or taxpayers in terms of cost savings from the project must not only be enough to repay the investor with interest but must also be large enough to cover the significant transaction costs including the costs of the evaluation itself. Hence the financing mechanism imposes a requirement that evaluation costs must be kept low in order for the PFS initiative to be successful.

This paper examines the consequences of this requirement for decisions regarding the choice of an appropriate evaluation design and even the choice of services funded. This research starts by setting up the analytical framework describing the private investor's decision to engage in a social impact financing arrangement and demonstrates numerically that the magnitude of evaluation costs negatively affects the probability that a PFS project will be successful. The next step involves scanning the existing set of PFS contracts currently in operation in the United States and abroad, where detailed information of the evaluation process is available, to understand the types of study designs used and the choice of outcomes measured to analyze the available information on PFS-related evaluation costs.

We discuss whether inexpensive impact evaluation frameworks such as designs lacking a comparison group may have merit especially in the context of advances in PFS in Europe using a rate card approach. We argue that while social impact financing offers the promise of expanding public sector reliance on interventions that work, the need to minimize evaluation costs implies that valuable efforts in process evaluation will be underfunded. Hence through PFS, while we may learn which interventions work and do not work but not why, and how the service could be improved to produce desirable outcomes. We further discuss whether encouraging rather than discouraging process evaluation in PFS would result in better social outcomes in the long-run.

This paper is directly linked to the APPAM 2018 theme of “Evidence for Action” as it investigates the role of evaluation in expanding the scope and improving the delivery of social programs through an innovative social impact financing model.