Association for Public Policy Analysis & Management

Panel: Producing Evidence in the Investing in Innovation (i3) Program: Supporting Rigorous Evaluations

(Education)

Background: Established in 2009 by the American Reinvestment and Recovery Act, the Investing in Innovation (i3) Program has awarded over $1.3 billion to local education agencies and nonprofits to implement and evaluate a wide range of interventions in three tiers. Scale-up grants were awarded to support taking interventions with strong evidence of effectiveness to national scale, Validation grants support the validation of effectiveness of funded interventions, and Development grants support grantees to establish the feasibility of the intervention’s implementation and test its promise. A goal of the i3 Program is for the grantees to produce rigorous evidence of the impact of the funded interventions on student achievement; the i3 Program funded the provision of an array of supports aimed at increasing the likelihood that the evaluations would make a substantial contribution to the evidence base about what works for schools, teachers, and students. Objectives: This panel will discuss lessoned learned from i3 about how to support and realize the goal of producing rigorous evidence in a tiered evidence program. The first paper will describe how i3, in partnership with the Institute of Education Sciences, made rigorous evidence a priority, made resources and supports available to grantees and third-party evaluators, and supported grantee/evaluator partnerships with the goal of producing rigorous evidence. This paper will describe the type, number, and cost of these resources, and present results from detailed reviews of the strength of the impact and implementation evaluations conducted with i3 funds. The second paper will discuss, from the perspective of a grantee organization, the effect of i3 funding and prioritization of rigorous evidence on the organization. It will track the journey of the organization into the era of ‘evidence-based’ grant and policy making, and describe how the organization has grown and evolved to respond to increasing demand for evidence of program effectiveness. The paper will discuss how i3 affected the organization’s capacity to integrate evaluation into the development and delivery of the program, form partnerships with evaluators, and engage policymakers in using the evidence in their decision making. The third paper will discuss, from the perspective of an evaluator, how being a 3rd party evaluator on several i3 grants for the same grantee organization has supported that partnership, and how their relationship and use of evidence has evolved. It will give examples of how the evaluation resources provided by the program supported both the evaluation and the evaluator/grantee partnership, and how the grantee has been able to continually engage in the production of rigorous evidence at increasing scale. Discussion: The discussants will connect these examples from this particular tiered evidence grant program to the wider policy context in which grant applicants are increasingly asked to provide evidence of effectiveness and grants are awarded with the expectation that additional evidence will be produced. They will provide insights into how the evidence from these efforts can be used in policymaking, and identify potential gaps between what policy makers need and what these evaluations provide.