Many of the most compelling educational interventions now being developed – especially those designed to address the urgent needs of schools serving low income and underachieving students – involve supporting educators to fundamentally change their everyday practices.  One example is the upcoming implementation of the Common Core State Standards in the vast majority of school systems in this country. 

Measuring the quality of these interventions’ implementation is not a straight forward matter. To capture the inherent multi-dimensional and interactive nature of implementation quality in these interventions, do we (a) import high levels of expert judgment and extract high-level qualitative and quantitative assessments of implementation quality; or, (b) granularize these complex sets of behaviors into indicators that can be more easily and perhaps reliably rated by less thoroughly expert observers but potentially sacrifice face and predictive validity by doing so?

In the Implementation Quality Rating System, IRRE has taken a novel approach to resolving this measurement dilemma. This paper presents this approach and some initial psychometric results from its application to assessing the implementation quality of a curricular intervention in high schools serving low income students. The system is organized as a vertically integrated set of constructs and measures.  Implementation quality for an intervention is first parsed theoretically into broad elements of implementation quality – what are the “active ingredients” of this intervention (in this case, implementation of a thematic curriculum and its associated instructional strategies).  Next, nuggets of effective practice are specified within each of these elements.  For example, formative assessments might be an element of implementation quality and all students held individually accountable on these assessments might be one nugget of effective practice for this element. The next step is to calculate density indices with respect to these nuggets of effective practice – how many of these nuggets were reported/observed for each of the quality elements within and across each instance of this element.  These indices are then used to obtain quality ratings of each implementation element.  Finally, the system specifies combinatory rules for integrating these implementation element ratings into an overall rating of implementation quality for a given intervention.

This approach is supported by a data collection, analysis and reporting software package  that produces immediate and continuously updated reports that range from: presence or absence of a specific “nugget of effective practice” in the three instances of formative assessment during a teacher’s lesson; to a snapshot or growth report of the quality of formative assessments in all science teachers’ classes in a school; to a district wide rating of the overall quality of implementation of that science curriculum in all high schools.

This paper will provide greater detail on the steps in this approach to measuring implementation quality and initial psychometric information on the tool from its early application to curricular interventions. How this approach might advance research and evaluation on educational interventions currently being planned and launched as well as potential challenges and limitations of this tool as a research and evaluation instrument will also be discussed.