Association for Public Policy Analysis & Management

The Investing in Innovation (i3) Fund awarded competitive grants to school districts and nonprofits to implement and evaluate educational interventions with varied prior evidence of effectiveness. The i3 program aims to build the evidence base on effective educational interventions. The short-term goal of the i3 program is therefore strong evaluations that provide high-quality data on implementation fidelity and independent impact evaluations that meet What Works Clearinghouse™ (WWC) Standards and adequately represent the population served by the intervention. The long-term goal of i3 is to identify effective interventions that can be implemented with adequate fidelity at increasing scale.

This paper summarizes the strength of and findings from the 67 i3 evaluations completed by May 2017 (39% of the 172 grants awarded to date). The 67 grants were awarded a total of $679 million, or 49 percent of the total i3 investment to date. The sample includes four Scale-up grants (36 percent of the 11 awarded to date), 15 Validation grants (33 percent of the 46 awarded to date) and 48 Development grants, (42 percent of the 115 awarded to date).

To meet the short-term goal of i3, the i3 evaluations were expected to produce high-quality implementation data, be independent, meet WWC standards, and be adequately representative of those served by the intervention. Almost all of the i3 evaluations (63 of 67, or 94 percent) conducted a high-quality implementation evaluation and almost all of the impact evaluations were considered independent. Almost three-quarters of the i3 evaluations (48 of 67, or 72 percent) also included strong impact evaluations that meet WWC Standards. Sixty percent of the i3 evaluations (40 of 67) included a sample that adequately represented those served by the intervention, thereby meeting the short-term goal of i3. This includes one of the four Scale-up grants, 13 of the 15 Validation grants, and 26 of the 48 Development grants.

The long-term goal of i3 is to build evidence that identifies interventions that were implemented with adequate fidelity and have positive impacts on student academic outcomes. Almost 80 percent of the interventions were implemented with adequate fidelity. And 12 (18 percent) of the interventions had a positive and statistically significant impact on at least one student academic outcome. Thirteen percent (9 of 67) of the interventions were implemented with fidelity and had produced positive impacts. This includes two Scale-up evaluations, four Validation evaluations, and three Development evaluations.

The percentage of i3 evaluations that met the long-term goal of i3 differed by grant type: 50 percent of the Scale-up and 27 percent of the Validation grants met the long-term goal of i3, compared with 6 percent of the Development grants. Perhaps this is not surprising given the high expectations for prior evidence that grant applicants needed to meet to receive a Scale-up or Validation grant.