Association for Public Policy Analysis & Management

A majority of states and many school districts have implemented principal evaluation systems in the last ten years with a desire to identify effective/ineffective principals and help principals grow professionally. As a whole, these evaluation systems are not tested for validity or reliability, so we have no evidence that these evaluation systems are an accurate measure of principal effectiveness. We suggest an alternative principal effectiveness measurement model that combines superintendent ratings, survey-based measures, and objective measures of principal effectiveness into a three factor model where principals are evaluated based on their instructional, strategic, and human resource leadership skills.

We validate this model using data from North Carolina public schools with a confirmatory factor analysis multitrait-multimethod (CFA-MTMM) approach. In this approach, the latent traits are estimated through the different available measures using CFA. The shared method variance is accounted for by allowing the disturbances of the variables that are measured the same way to correlate with each other. The key advantage of testing CFA-MTMM is that modeling CFA produces quantified measures of model fit. In this context, a well-fitting model means that the specified CFA structure is a relatively accurate approximation of the original data. In this way, CFA-MTMM models that have good fit indicate that the measures are accurately matched with the traits that we are hoping to measure. Once we have a well-fitting CFA-MTMM model, we can then explore the convergent validity between our different measures of the same principal effectiveness constructs.

Each of the three traits (instructional, strategic, and human resource leadership) is matched with five different indicators. Of the five indicators, one is the superintendent rating of that specific leadership skill, two are objective measures of the trait, and two are survey measures of the trait. The objective measures include teacher and school level value added measures, teacher evaluation scores, school-level test score passing rates, and teacher turnover information. The survey measures come from the Omnibus survey, a survey administered to a stratified random sample of schools in North Carolina as part of the Race to the Top grant in the school years 2011-12 through 2013-14.  We select multiple indicators for each latent trait with repeated types of indicators within a trait in order to effectively separate measurement error from the factor loadings of each indicator.

We find a CFA-MTMM model with multiple indicators of good model fit and statistically significant, moderately-sized factor loadings, providing evidence of convergent and construct validity. The factor loadings indicate that the variables most strongly associated with each factor are measured using different methods depending on the principal leadership trait: for instructional leadership, the highest factor loading is on the average teacher value added score in the school; for strategic leadership, the highest factor loading is on the percentage of students in the school who pass the state assessment; and for human resource leadership, the highest factor loading is on an Omnibus survey scale measuring common purpose  (shared beliefs and purpose).