Association for Public Policy Analysis & Management

Panel: Doing Cost-Benefit and Return-on-Investment Analysis Better

(Methods and Tools of Analysis)

Cost-benefit analysis (CBA) and return-on-investment (ROI) analysis are critical tools for evidence-based policymaking by federal and state governments. Evidence on impacts provide only half of the picture to policymakers who need to understand whether the effectiveness of a policy is worth its cost. This information is even more valuable in tight budget environments and a policy space with many competing priorities. Despite the unambiguous importance of CBA/ROI analyses, researchers continue to disagree on the most appropriate CBA/ROI techniques. Even when there is agreement on methods, the limitations of pre-existing administrative data sources and other obstacles may prevent researchers from implementing their ideal approach and force them to search for an acceptable second-best solution.

Each panelist will discuss these challenges, drawing on presenters’ experiences with CBA/ROI analyses of policies ranging from internet privacy regulations to home visiting programs to integrated career pathway programs. Some of the challenges that will be discussed include the problem of monetizing benefits, including persuasive measurement of willingness to pay, difficult to estimate social benefits, and the projection of benefits into the future. Researchers often must conclude a study long before all the benefits of a policy are fully realized. In such circumstances, the results of a CBA/ROI can be strongly affected by the chosen approach to projecting those benefits, including whether benefits will decay, persist, or whether the largest benefits are even delayed far in to the future. Sometimes an existing literature provides estimates of benefits or inputs to equations for calculating a benefit, but practitioners often struggle with how to map estimates from the literature onto their own studies in a reasonable way. On the cost side of a CBA/ROI, some of the most important costs are often the wages and salaries of program staff who undertake programming in combination with other organizational responsibilities. It is not always clear to researchers how to assign the time of these staff to the intervention, or how to understand what similar services they may be providing in a counterfactual world. Costs and benefits are affected by the choice of the discount rate, which is still the subject of well-choreographed debate. The arguments for and against a relatively high or low discount rate are well understood, but researchers have not reached a consensus – particularly for analyses of policies with long-lasting or catastrophic effects.

Each panelist will discuss how they navigated these questions in their own studies, and the importance of sensitivity testing and methodological transparency for doing CBA/ROI analysis better. Attendees will be invited to take part in a discussion of how to move forward CBA/ROI methodology as a field.