Association for Public Policy Analysis & Management

A growing body of research about school climate and conditions for learning tells a clear story: The climate in a school and the conditions in which students learn matter. In their review of school climate research, Thapa et al. (2013) show that a positive school climate is associated with reduced risk behavior, aggression and violence, absenteeism, and suspension; and is associated with increased motivation to learn and academic outcomes. The Institute of Education Sciences highlights the importance of positive school climate for dropout prevention, noting that caring, supportive environments coupled with high expectations for student achievement are associated with improved student outcomes (Dynarski et al., 2008). Osher et al. (2008) explores the connection between school climate, the conditions for learning, and academic outcomes, noting that students experiencing stress will have difficulty attending to learning, and that teachers need to find the balance between challenging and supporting their students to achieve the best outcomes.

As standardized measures of school climate become more, the use of such measures in rigorous evaluations of educational interventions is expected to grow. Conducting rigorous research requires understanding the statistical properties of the outcome measures. As cluster randomized trials gained prevalence in educational research, Hedges and Hedberg (2007) published a widely cited article examining the intraclass correlation of reading and mathematics scores. Their findings have proven invaluable to researchers who are planning studies by providing necessary estimates of the intraclass correlation of standardized assessments. The objective of this paper is to provide similar estimates of the ICC and other variance decompositions for measures of school climate that can be used to plan studies of interventions aimed at improving school climate.

The Conditions for Learning (CFL) survey is a psychometrically validated survey instrument used to measure the conditions for learning at the elementary school (grades 2–4), middle school (grades 5–8), and high school (grades 9–12) levels. The CFL survey is designed to assess the four core conditions for learning constructs identified by Osher et a. (2008): (1) a safe and respectful climate, (2) challenge/high expectations, (3) student support, and (4) social and emotional learning. Cleveland Municipal School District began administering the CFL in 2007-08 in the spring, and continued to do so annually through 2011-12. Starting in 2012-13, the CFL was administered three times annually. This study uses data from students in all schools in CMSD, measured one to three times annually over nine years.

Using these data, we will estimate the intraclass correlation, cohort-to-cohort variance (within school over time), and group-level reliability (Raudenbush & Sampson, 1999) by modeling the data using mixed-effect models. Estimates of the ICC for each grade band and each construct can be used in power analysis for designing cluster randomized trials. Within school cohort-level variance can be used in power analysis for comparative interrupted time series models. Group-level reliability can be used to determine the extent to which changes in school climate can be detected both within a single school over time, or between schools.