Association for Public Policy Analysis & Management

It is not only challenging to understand how programs work, but also to translate such evidence into policy. In a recent meta-evaluation of 108 Health Care Innovation Awards (HCIA, Round 1) funded by the Centers for Medicare and Medicaid Innovation, we tried to address this challenge with several innovations. In addition to analyzing the impact of each intervention, we used contextual and qualitative data to filter interventions, allowing a quick, visual, interactive understanding of results related to specific features of interest. We used meta-analysis to synthesize quantitative evaluation findings across the HCIA programs to assess four outcomes: total cost, hospital admissions, hospital readmissions, and emergency department visits. We also systematically collected information on the implementation characteristics and features of the HCIA innovations.

To make the information even more actionable in a policy context, we created a Bayesian data dashboard that highlights the benefit of interpreting findings in an intuitive, probabilistic way. For each intervention, we present the Bayesian posterior distributions for the outcomes of interest with sliders and interactive text. Together this enables the users to bring their own questions, such as “What is the probability that costs were reduced by $10 or more?” and supports policy makers in applying their own risk tolerance to the finding. This type of analysis is in stark contrast to traditional presentations of analyses that rely on static questions with static answers and stacks of regression tables. We provide a demonstration of the tool, which is built with open source software and easily adaptable for other applications.

While we developed the Bayesian dashboard for the meta-evaluation of the HCIA programs, it demonstrates how Bayesian analyses can be presented in an engaging and understandable way, and it shows how Bayesian findings can be integrated with programmatic features data derived from traditional methods. Available as a web-based or portable app, the dashboard provides a flexible framework for how Bayesian program evaluation results can be presented in the future. Most importantly, it provides policymakers a bridge from program research findings to understandable evidence that can be used to assess and improve healthcare programs.