Association for Public Policy Analysis & Management

Impact evaluations in the social sciences that causally estimate effects by using randomized experiments have become the accepted “gold standard” of evidence (US Department of Education, 2003). While causal impacts are necessary, evaluations without an analysis of costs are not sufficient for policymaking (Ross, Barkaoui, & Scott, 2007; Monk, 1995). However, costs have not historically been included in evaluations in a systematic, rigorous way (Rice, J.K., 1997; Levin, 2001; Clune, 2002; Harris, 2009; Levin, 2013).

Recently, the Department of Education, Department of Labor, USAID, and other major sources for research funding have requested that new research proposals include an economic evaluation in the form of a cost study, cost-effectiveness analysis, or benefit-cost analysis.

Many studies or reports of costs rely on budgetary data or the quoted price for a program that is available for sale. Budgets are designed to monitor expenditures rather than to estimate efficiency (Coombs & Hallak, 1987: pg. 46). Instead, budgets reflect expenditures for a particular time period, mask quantity from price, lack important descriptive information (personnel), and do not include resources previously purchased (facilities, computers) or those that were donated (volunteer time, donated materials). Given that replication and scale up are often aims of evaluating programs, a cost estimate based on budget data would not reflect the needed ingredients and the true cost of the program to accomplish that goal.

The ingredients method was developed to provide a straightforward method to conduct economic analyses in education and other public sectors (Levin, 1975; Levin, 2001; Levin, 2013). The ingredients method outlines and describes all of the ingredients used to implement a policy or program to address two primary questions: 1) What is the total social cost of the program? 2) How are the costs financed? The method is consistent with both the economic concept of opportunity cost and standard practices for cost accounting (Levin, 1975). In addition, the ingredients method provides a systematic approach to assessing the cost of each ingredient and calculating the cost per student so that interventions can be compared for their relative efficiency (Levin & Belfield, 2015).

While the ingredients method is widely accepted as a rigorous approach to evaluating costs, specific guidance for field trials is needed regarding designing and conducting rigorous economic evaluations. This paper addresses this need by presenting 10 steps to improve the methods of cost analysis used in experiments to examine the cost of social interventions.