Association for Public Policy Analysis & Management

Peer review at the National Institutes of Health has been criticized for being overly subjective (Pier et al 2018). In this paper, we examine whether peer review actually serves as a reliable guide for government agencies to select scientific ideas that have the potential to become the best science. We use data from the Ruth L. Kirschstein National Research Service Award (NRSA) at NIH to evaluate whether peer review or program officer discretion identifies the best science. Unlike research grant programs at NIH, program officers in charge of the NRSA training program use significant levels of discretion to determine NRSA fellowship awards (Heggeness, Ginther, Larenas & Carter-Johnson, 2018). We compare the career outcomes for NRSA applicants who were awarded the fellowship based on peer review rankings or by the discretion by the program and institute staff to determine which method is better at predicting who will succeed in a future NIH-funded independent research career. Our preliminary results indicate that the NIH peer review system does better at predicting who will become engaged in future independent scientific research than program officer discretion.

References:

Heggeness, M.L., Ginther, D.K., Larenas, M.I., Carter-Johnson, F.D. (2018). The Impact of Postdoctoral Fellowships on a Future Independent Career in Federally Funded Biomedical Research. Working Paper, University of Kansas.

Elizabeth L. Pier, Markus Brauer, Amarette Filut, Anna Kaatz, Joshua Raclaw, Mitchell J. Nathan, Cecilia E. Ford, Molly Carnes. (2018). Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences Mar 2018, 201714379; DOI: 10.1073/pnas.1714379115.