Association for Public Policy Analysis & Management

In the spring of 2014, the School District of Philadelphia (SDP) launched a new District-wide survey program for students, teachers, principals, and parents and guardians. In collaboration with University of Pennsylvania’s Graduate School of Education, staff from SDP’s Office of Research and Evaluation (ORE) developed the surveys using the research of Bryk and colleagues (2010) on the five essential supports for school improvement: leadership, professional capacity, climate, instruction, and parent/guardian community ties. Each construct is comprised of multiple sub-constructs, which vary across each of the four surveys. For example, the Climate construct includes the following sub-constructs for students: Belonging, Bullying, and Safety/building conditions.

Bryk et al. (2010) found that schools that were strong in at least three of the five essential supports, based on surveys, saw the greatest improvements in achievement, whereas schools with low scores on one or more indicator had less than a ten percent probability of improving. Building on this work, ORE tested relationships between its survey data and standardized test scores. Essentially, ORE wanted to know: do Philadelphia’s surveys relate to student outcomes as is the case in Chicago?

For the student, teacher, and parent/guardian surveys,[1]ORE created construct scores for each school that met the survey response threshold. These scores were based on the percentage of respondents that selected the most positive responses to questions within each sub-construct. The sub-construct scores for each survey were then averaged to create an overall construct score for the parent/guardian, student, and teacher surveys. School-level construct scores reflect an average of the scores from the three respondent groups. For example, a school’s Climate score is the average of the parent/guardian, student, and teacher Climate scores.

Initial correlation analyses examined the relationships between schools’ construct scores and state standardized test scores. Results from correlations using 2015-2016 school year survey data confirmed the theoretical framework of the surveys in that there where strong correlations between school-level survey construct scores and the school’s average student score on Pennsylvania standardized tests (PSSAs for grades 3-8 and Keystones for Algebra 1 and English). In particular, Climate construct scores from the student, teacher, and parent/guardian surveys were significantly correlated with the average PSSA math and English scores.

Analysis of the relationship between Instruction construct scores and standardized test scores yielded similar results, with highly significant correlation among all surveys and all test scores. Further analyses delve into the particular nuances captured by the sub-constructs; for example, one finding is that one sub-construct under teacher Instruction (“student engagement”) is more highly correlated with PSSA scores compared to the second sub-construct (“self-reflection”). In our paper, ORE will include findings from these analyses on the Climate and Instruction constructs and sub-constructs, as initial correlations suggest these two constructs have the strongest relationship to test scores. The paper will also discuss the ways that these findings inform ORE’s work and the work of school improvement across the District.