Association for Public Policy Analysis & Management

In recent years, the pressure has mounted on organizations to demonstrate rigorously the impact of their work—to provide evidence that their programs make the changes they promise. A confluence of factors has contributed to the push for impact. Huge reductions in the cost of gathering and analyzing data have been accompanied a global trend toward results-oriented funding and management. The culture of giving philanthropic sector has shifted as well, with wealth shifting to younger donors with a strong desire to make investments with demonstrated impact. The expansion of available data is also accompanied by calls for more accountability in the nonprofit sector. The demand for impact data has accelerated further with the focus on pay-for-performance and social impact bonds.

While the pressure to measure impact has never been stronger, many nonprofits are not well placed to engage in randomized evaluations and other impact evaluation strategies. The timing for an impact evaluation is not right, resources or sample size are insufficient, the logistics make randomization infeasible, or the question to be answered simply isn’t a question that a randomized evaluation helps to answer. Yet the desire to measure impact often leads organizations in one of two dangerous directions: collecting mountains of data that cannot be used to measure impact or collecting insufficient data to demonstrate accountability and to learn what to do in the future.

How should nonprofits responded to these pressures? The conventional wisdom suggests that organizations should make greater investments in impact data, working towards impact measurement. This paper challenge this wisdom. The Goldilocks Challenge takes aim at the idea that organizations should always seek to measure their impact, presenting an alternative strategy that helps organizations consider when to gather outcome data and measure impact, and when not to do so. This strategy, called the CART, is based on the work of the Goldilocks Initiative. The CART - Credible, Actionable, Responsible and Transportable data collection - is a set of principles organizations can use to identify the right time to engage in impact evaluation, and – just as importantly – build systems that provide information that supports learning and improvement.

This paper argues that pulling back from a focus on impact will allow both social sector organizations and their funders to reconsider the push for measuring impact to focus instead on learning and improvement. We present five kinds of non-impact data that can drive better outcomes. Data from ten case studies demonstrate the ability of this approach to provide data that improves social outcomes. Cases are drawn from traditional social services nonprofits, social enterprises, and NGOs, and include both traditional contracting, foundation-nonprofit relationships and pay-for-performance programs.