Intraclass Correlations and other Variance Estimates for Designing School-Climate Research Studies
*Names in bold indicate Presenter
As standardized measures of school climate become more, the use of such measures in rigorous evaluations of educational interventions is expected to grow. Conducting rigorous research requires understanding the statistical properties of the outcome measures. As cluster randomized trials gained prevalence in educational research, Hedges and Hedberg (2007) published a widely cited article examining the intraclass correlation of reading and mathematics scores. Their findings have proven invaluable to researchers who are planning studies by providing necessary estimates of the intraclass correlation of standardized assessments. The objective of this paper is to provide similar estimates of the ICC and other variance decompositions for measures of school climate that can be used to plan studies of interventions aimed at improving school climate.
The Conditions for Learning (CFL) survey is a psychometrically validated survey instrument used to measure the conditions for learning at the elementary school (grades 2–4), middle school (grades 5–8), and high school (grades 9–12) levels. The CFL survey is designed to assess the four core conditions for learning constructs identified by Osher et a. (2008): (1) a safe and respectful climate, (2) challenge/high expectations, (3) student support, and (4) social and emotional learning. Cleveland Municipal School District began administering the CFL in 2007-08 in the spring, and continued to do so annually through 2011-12. Starting in 2012-13, the CFL was administered three times annually. This study uses data from students in all schools in CMSD, measured one to three times annually over nine years.
Using these data, we will estimate the intraclass correlation, cohort-to-cohort variance (within school over time), and group-level reliability (Raudenbush & Sampson, 1999) by modeling the data using mixed-effect models. Estimates of the ICC for each grade band and each construct can be used in power analysis for designing cluster randomized trials. Within school cohort-level variance can be used in power analysis for comparative interrupted time series models. Group-level reliability can be used to determine the extent to which changes in school climate can be detected both within a single school over time, or between schools.