A series of evaluations studied different methods to help struggling secondary school readers improve their skills.   The evaluations all confronted the issue of how to describe and quantify implementation measures for programs which can be quite complex.   The paper explores how this was tackled in three different evaluations, which approached the problem in different ways.   The first program sought to improve academic achievement of struggling readers in eight middle schools in a large urban district by providing two-year school wide professional development for middle school content area teachers to support their integrated use of reading strategies. Using one of three dimensions comprising the Concerns-Based Adoption Model (Hall et al., 2006), an innovation configuration (IC) map was developed to construct a common understanding of the innovation and of variations likely to be observed in classrooms (Hall & George, 2000; Hall & Hord, 2006). The IC map explicated targeted approaches thereby distinguishing them from among a range of commonly used practices. Coaches collaborated with researchers to collect implementation data and assign IC map ratings based on their ongoing observations and documentation of implementation.  After considering benefits and costs to developers, implementers and researchers of using this approach, resources were invested to design an observation protocol aligned with the IC map and provide evaluators with intensive training to collect observation data and assign fidelity ratings.  Findings from a related study of inter-rater reliability led to revisions in materials used to train evaluators.  The IC map was also used by developers to refine the PD and aligned with protocols used by coaches to support classroom

Two other programs sought to help struggling secondary school readers by implementing the Voyager Passport Reading Journeys program. The VSRII project involved eight middle schools in three urban school districts, while the ISR project involved six high schools in four rural and suburban districts. Voyager Passport Reading Journeys (VPRJ) is a supplemental reading program for adolescent students reading below grade level that is highly structured and blends teacher-led targeted instruction with student-centered technology. The VPRJ structure provided a relatively straightforward framework to develop the research questions and data collection instruments. The implementation study included classroom observations and monthly surveys of each teacher and interviews with the project director and each Cambium VPRJ coach assigned to each school. Using Cambium’s Implementation of Fidelity Index as a starting point, we developed a rubric that closely followed the VPRJ lesson structure. The resulting rubric included four components: classroom environment, quality and amount of instruction, use of differentiation strategies, and classroom management. Challenges encountered included differences in classroom space and available instruction time across schools, differing accounts of what coaches said were “appropriate” modifications to the intervention, and differing requirements for training and coaching (since coaching was sold by the day).