Association for Public Policy Analysis & Management

When a new mayor and a new school chancellor took office in 2014, the New York City Department of Education (NYC DOE) sought to redesign their annual student, parent, and teacher school climate surveys. Inspired by the Chicago Consortium’s book Organizing Schools for Improvement (Bryk et al., 2010), NYC DOE leadership used research to identify aspects of school climate that are associated with school improvement and then planned to measure them with their redesigned surveys. In turn, information from the redesigned surveys would describe schools’ strengths and weaknesses and provide schools with target areas in need of improvement.

The redesign effort included a collaboration with the Research Alliance for New York City Schools in 2014 (RANYCS). Together, NYC DOE and RANYCS began with Chicago’s full list of survey measures, and then adapted them to the NYC context. To determine which concepts we should add, we looked to NYC DOE staff, as well as expertise from the Research Alliance. The NYC DOE wanted to include measures that were aligned with the administration’s emerging theory of school improvement, including the importance of providing guidance for next-level readiness, teaching social-emotional learning skills, creating an effective School Leadership Team,[1] promoting inclusive classrooms, and teaching aligned to the Common Core’s. The Research Alliance proposed additional concepts drawing on our own research as well as the broader field. Where possible we depended upon previously validated measures, and created our own using our collaborative expertise when necessary.

Today, the NYC School Survey is the largest education census in the United States. The Survey is administered annually in more than 1,800 schools to all teachers and parents in the system, as well as all students in grades 6-12. In the spring of 2015, 66,957 teachers, 421,160 parents, and 401,628 students took the Survey.

In this paper, we assessed the redesigned survey measures on their reliability, within-school agreement, and face and predictive validity. With individual data for the 2014-15 survey and administrative records, we find that most of the student and teacher measures have strong psychometric properties, and we will use that information to discuss how these measures could be analyzed to inform policy decisions.

To inform policy decisions about individual schools such as accountability or resource distribution, these measures should be both predictive of student outcomes and be able to meaningfully distinguish between schools. In a multi-level framework, we model individual survey results from 2014-15 and 2015-16 with individual level administrative data from 2013-2016 to analyze how well school climate estimates predict future student achievement and how well they distinguish between schools. The findings will be presented in the proposed presentation to inform a discussion of the benefits and drawbacks to using climate measures to inform policy decisions.