Association for Public Policy Analysis & Management

Economists often face criticism that their theoretical and empirical models lack validity due to the large number of simplifying assumptions necessary for models to be analytically tractable. The economic evaluation of educational policies and programs employing the ingredients method for cost, cost-effectiveness, or cost-benefit analysis is no exception to this critique. Educational economists must make assumptions due to two sources of uncertainty: model uncertainty, as in the well-documented debate (see, e.g., Boardman et al., 2011, Chapter 10) over the selection of the appropriate social discount rate to calculate present value, and empirical uncertainty due to the infeasibility of gathering sufficiently detailed data on all resources. The assumptions necessary in the latter category include the selection of an appropriate market or shadow price to represent the value of a good or service in opportunity cost terms, the amount of time a resource is available for use to divide it among shared uses, the time horizon over which to annualize the cost of physical and human capital resources, and the sample of participants over which to calculate an average cost in the presence of attrition, among others.

This paper proposes a set of harmonized assumptions that can be used to increase comparability of economic evaluation across programs and across studies. By building consensus on a set of reasonable, empirically derived assumptions that are selected so as to minimally distort the results of evaluations, differences in costs, cost-effectiveness, and benefit-cost ratios can be more confidently ascribed to meaningful differences in resource use, program implementation, and program effectiveness, as opposed to differences in choices made by the analyst. The paper also tests the robustness of results to the assumptions used in order to identify which assumptions most contribute to sensitivity of results and thus merit extensive sensitivity analysis. Finally, the paper aims to reduce the efforts needed to collect data for the ingredients method by suggesting where simplifying assumptions can be made without substantially distorting results, thereby enabling the use of existing survey, observation, or other data sources for rigorous cost analysis.

We apply three methods, each using separate sources of data. First, we synthesize results across a variety of existing surveys and studies (such as surveys of average school facility sizes in school construction trade publications) to obtain estimates of current consensus on appropriate values for each of these assumptions. Second, where consensus does not exist on reasonable values, we survey practitioners in various domains of education to obtain suggested mean and extreme values. Finally, using data from a large sample of cost studies we have performed across a variety of topic areas and domains (early childhood, K-12, and higher education, and in areas such as early literacy, social and emotional learning, and community-based student support programs), we run a series of simulations using Monte Carlo and break-even analyses to determine the probability of observing values extreme enough to substantively alter conclusions to ultimately determine to which assumptions and under what conditions results are robust or sensitive.