Association for Public Policy Analysis & Management

Panel: Expanding the Use of School Climate Surveys: Critical Considerations for Measurement and Policy

(Education)

The newest federal education legislation, the Every Student Succeeds Act, has spurred a focus on using school-quality metrics for state report cards and school accountability. As such, districts and states have broadened school quality definitions to include new metrics, such as school climate surveys. This proposed panel focuses on three efforts to adapt an existing school climate survey for new populations and highlights the critical step of ensuring these surveys function properly and provide meaningful data in these new contexts.

When first developed, tools are often tested for validity and reliability, at times focused only on a single population or setting. These “tested” measures are then frequently adopted by others without attention to how well they function within new contexts. The potential detriments of applying one tool in a new context without attending to possible variation in validity and measurement properties can be serious, especially when used for accountability purposes. The three proposed panel papers illustrate ways of testing the reliability and validity of surveys as they are implemented in new settings.

In the early 1990s, researchers at the Consortium on Chicago School Research developed a framework of organizational conditions necessary for improvement in student outcomes. Using a set of student and teachers surveys, now known as the 5Essentials, they measured those conditions schools. With those surveys now in use for more than two decades, research in Chicago shows them to be reliable and valid at both the elementary and high school level (Bryk, Sebring, Allensworth, Luppescu, & Easton, 2010; Sebastian & Allensworth, 2012).

This proposed panel brings together researchers and practitioners who are studying adaptations of the original 5Essentials surveys.  Two papers focus on findings from districts outside of Chicago: New York City and Philadelphia.  Both locales relied heavily on the 5Essentials as they implemented their own district-wide surveys, but capitalized on researcher-practitioner relationships to make modifications that responded to their local context and needs. The third paper presents an adaptation of the 5Essentials for use in school- and center-based early education settings.

All three papers focus on the process of adaptation for their particular populations, and the careful study of survey reliability and validity in these new contexts.  Presentations will also include implications for using these surveys in an accountability framework.  If school climate surveys are to be used as an ongoing, sometimes high-stakes measure of school quality, it is imperative that we understand whether, and in which ways, they function differently as a measure when applied somewhere new. It is also vital that we understand the benefits and potential risks of using these measures as school-quality accountability metrics. 

Dr. John Eason will serve as a discussant for this panel. He is one of the creators of the original 5Essentials, former Director of U.S. Department of Education, Institute of Education Sciences, and now at the Spencer Foundation. His unique perspective will highlight the importance of studying ongoing adaptations of original tools, and how this work fits within the current educational policy and reform context.