Motivation: Informal science education institutions, such as museums, zoos, and botanical gardens, are critical partners in public science education and support the education efforts of public schools. Despite the growing consensus that collaboration between formal and informal organizations is an important tool for improving science education in the United States, there are relatively few examples of ongoing, intensive collaborations between schools and informal education institutions, and there has been little research on the impact of such partnerships.

Program: Urban Advantage (UA) is one of the few existing collaborations between a public school system and informal educational institutions in the nation. The program, which is now in its tenth year, is an ongoing collaboration between the New York City Department of Education and eight informal science education institutions across NYC. UA harnesses the resources and expertise of NYC’s informal science institutions to: a) enhance the science content knowledge of middle school science teachers; b) develop teachers’ skills at using inquiry-based approaches in their classrooms; and c) improve the science achievement of middle school students. The program has grown to scale in NYC, and in 2013-14 it involved almost one-third of the district’s middle schools.

As the UA program has continued to evolve, IESP has conducted numerous evaluations, both quantitative and mixed methods, to explore whether UA improves student achievement in science, why it may work, and in what contexts it may work best. A previous study found a positive effect of attending a UA school on science achievement, but new data allows for a more in-depth, longer-term analysis of the program.

Methods: In this study, we will estimate the impact of attending a UA school in eighth grade on eighth-grade science test scores. We will use a difference-in-differences framework with school fixed effects to estimate a causal impact, exploiting variation within schools over time. That is, we will compare the outcomes of students in the same schools who attend the school when the school is and is not participating in UA.

This study improves on our prior impact estimates in two key ways. First, we examine the impact over a longer time period, which provides more variation in treatment and additional power. Specifically, we use a 10-year longitudinal student-level dataset, beginning in 2003-04 (the year before UA was implemented) and continuing through the 2012-13 academic year.

Additionally, our measures of UA participation are more nuanced, as we have more accurate program participation data that tracks school participation in UA over time and importantly provides the number of teachers within a school that are actively participating in the program. This will allow us to determine how program effects vary with respect to the concentration of teachers in the school who are involved in UA, which is one measure of program implementation.

Policy Implications: Results from this study have implications for districts and states that are seeking to improve science education. Specifically, we provide evidence about a formal-informal partnership that has been successful in improving science achievement in NYC middle schools.