Association for Public Policy Analysis & Management

This paper will describe the National Science Foundation (NSF) INSPIRE initiative, which supports interdisciplinary and potentially transformative research, and the evaluation of the effectiveness of its grant selection process. The NSF INSPIRE initiative is different from other programs at NSF in that the applicants are required to obtain support from multiple directorates for the project to be considered, and the applications are reviewed by NSF staff rather than through an external panel of experts. Because of these unusual features, NSF leadership was interested in determining whether the initiative was designed to meet its intended goals.

Despite the recent proliferation of government programs that support high-risk/high-reward research, few studies have been conducted to evaluate their effectiveness. Further, the only published studies that we were able to identify examined the level of innovation of funded research several years after the grant award. In contrast, our study focused on the effectiveness of INSPIRE in soliciting and selecting more original projects. The study employed a quasi-experimental design, comparing INSPIRE participants to matched non-participants from other NSF-funded interdisciplinary and core co-funded programs.

To examine whether INSPIRE selected more transformative and interdisciplinary projects, we recruited a panel of experts across topics included in the research studies, who evaluated a set of brief research summaries derived from funded INSPIRE and comparison proposals. Each summary was scored by at least one expert, who did not know whether it came from an INSPIRE or comparison proposal. We were able to demonstrate significant differences between the INSPIRE and comparison summaries using a relatively small sample of 50 grant summaries. The INSPIRE summaries were scored significantly higher for high-risk and out-of-the-box/original characteristics. In contrast, the two sets of summaries received similar scores on the characteristics related to interdisciplinary. Our methodology can be adapted for any program that supports research and development projects, and we will share lessons learned about how the expert panel process can be further improved.

The study also surveyed the INSPIRE and comparison Principal Investigators (PIs) and found that the INSPIRE grantees rated their proposals as significantly more original, paradigm-shifting, high-risk, and interdisciplinary than the comparisons and that most had not attempted to fund these applications through more traditional merit review mechanisms, as they did not expect them to be funded. The INSPIRE and comparison PIs were similar on professional characteristics and publication history, suggesting that the program encouraged more innovative and interdisciplinary submissions.

During this time of decreasing federal budgets, many funders are searching for new ways to support research and development. We demonstrated that an unusual grant selection mechanism which relied exclusively on the internal agency review was effective in selecting high-risk, high-reward projects and developed and tested a strategy for analyzing the effectiveness of the selection process.