Association for Public Policy Analysis & Management

Roundtable: Forecasting the Effects of Public Policies on Budgets and People

(Tools of Analysis: Methods, Data, Informatics and Research Design)

Quantitative policy analysis is often used retrospectively to evaluate the consequences of a policy after it has been implemented. But it also plays an important role in providing guidance to policy makers as they consider potential policy alternatives. Although some policy options are preferred (or disliked) by legislators because of ideological or political considerations, many debates about policy center on empirical questions: How much will this option cost? How many people will win or lose? The importance of quantitative estimates of the costs and consequences of policy proposals has led to the establishment of dedicated forecasting agencies within budget offices (such as at the CBO and OMB, and in corresponding state budget offices). Working backward from the importance of final budget numbers, policy advocates and analysts often commission such forecasts from modelers outside government, to buttress policy proposals or assist legislators in designing effective proposals. Policy forecasts are often made using complex forecasting models, informed by parameter estimates drawn from evaluations of prior policy innovations. In some cases, as with the Affordable Care Act, multiple different public and private agencies make forecasts of the consequences of new legislation. A small literature examines forecasting techniques and assesses the strategies forecasters use to make predictions in different arenas. Perhaps surprisingly, however, there have been only a few assessments of how well forecasters performed in the past and how models might be improved to generate better forecasts in the future. Assessing the predictive power of forecasts is challenging. After a reform takes place, there is no control nation where business is carried on as usual. Political, social, economic, and further legislative changes confound the impact of any single initiative. Nonetheless, we are only likely to improve future forecasts through assessment and analysis of prior predictions. This roundtable of forecasters and policy analysts will take a critical look at the policy forecasting enterprise. The roundtable will look at forecasts of the Affordable Care Act, as well as other legislation. Participants will assess the predictive power of forecasts and discuss whether forecasters have a systematic tendency to over- or under-estimate the consequences of policy innovations.