Association for Public Policy Analysis & Management

The Institute of Education Sciences (IES) has funded over 300 efficacy studies of education programs and policies (Taylor & Doolittle, 2017). Randomized field trials have been a major focus for IES, as experimental evaluations are the accepted “gold standard” of evidence (US Department of Education, 2003). While causal impacts are necessary, evaluations without an analysis of costs are not sufficient for policymaking (Ross, Barkaoui, & Scott, 2007; Monk, 1995).

Recently, the Department of Education, Department of Labor, USAID, and other major sources for education research funding require the inclusion of an economic evaluation in the form of a cost study, cost-effectiveness analysis, or benefit-cost analysis. This paper addresses this new demand by providing guidance on how to design a cost study and examples of how data collection can be integrated into field trials.

A major challenge facing researchers is that costs have not been historically included in this work and the methods of assessing costs have been largely absent from pre-doctoral training (Rice, J.K., 1997; Levin, 2001; Clune, 2002; Harris, 2009; Levin, 2013). Thus, more work is needed to disseminate the methods of economic evaluation and to highlight standards of quality.

The ingredients method was developed to provide a straightforward method to conduct economic analyses in education and other public sectors (Levin, 1975; Levin, 2001; Levin, 2013). The ingredients method outlines and describes all of the ingredients used to implement a policy or program to address two primary questions: 1) What is the total social cost of the program? 2) How are the costs financed? While this method is widely accepted as a rigorous approach to evaluating costs, there are misconceptions in the field about the effort required for this research within larger evaluations. This paper provides guidance specific to the design of cost research within randomized trails and impact evaluations.

A second major challenge relates to the funding required to conduct research. As the federal budget for this research is likely not growing, it is important to simplify the design and steps in conducting cost research within field trials. Guidance and examples are included in this paper on integrating cost research into typically existing components of evaluations.

One such example includes an ongoing efficacy study of a kindergarten reading program where the research on costs is completely integrated into the evaluation. By including a focus on resource use, data on costs are available from the developer, the implementation of the program, examinations of fidelity, and observing mediators and heterogeneous impacts. As a result, the additive elements required for to assess the ingredients of the intervention are quite limited and should not add substantially to the work required to complete the research.