Association for Public Policy Analysis & Management

HEROES (Helping Early Readers Obtain Excellence in Special Education) is funded through a four-year i3 development project, beginning January 2016. The intervention centers on an instructional model based on strong theory designed to improve literacy outcomes for beginning readers whose disabilities require them to have an individualized education program (IEP) for reading. The goal of the project is to improve student achievement to such a level that these students can be moved off IEPs.

There was little direct evidence that HEROES would work for children with reading IEPS. There was evidence that the intervention worked well for struggling first-grade readers, but it was not known if the results would generalize to the targeted sub-population. Our initial plan was to randomly assign children to receive HEROES or not; barriers in recruiting districts led to our relinquishing the RCT for a quasi-experimental design. We are currently in our third year of data collection, and so far, the results reveal strong impact effects in favor of HEROES. We also are conducting an internal, formative evaluation to guide instructional decision-making.

Lessons Learned about Evidence Building From the i3 Project

Without the grant, there would be no evidence to support HEROES. The requirement to collect impact evidence made it possible to conduct multi-year studies in several districts across four states. Districts realize the funding is not possible without research and development activities, so the grant requirements provide the incentive to gather evidence.

We purposively created project goals that required the integration of evidence-building. One of our project goals was to conduct a rigorous external evaluation, which demands that we report annually on its progress. Another one of our key goals was to conduct an internal, natural variation study to identify which features work best in concert to improve students’ achievement levels, and then to use the results to revise the daily lesson format.

Sustaining Evidence Building Activities

As long as HEROES exists, we will collect evidence to guide program improvement and document its success. We purposively implemented HEROES in schools districts from the beginning to ensure that the instructional framework could be properly implemented in schools. Due to barriers in recruitment, however, we had to drop (for now) the RCT and implemented a quasi-experimental design instead. Our goal next year is to conduct lab-based experimental designs with targeted children both after school and during the summer to increase the internal validity of the evidence.

Besides distributing to school districts the yearly external evaluation findings, we produce district-specific annual reports that provide HEROES effect estimates and other program information based on the district data. We also provide in those reports the overall effects so districts have a comparison benchmark. Further, we developed on-line data logs for teachers so they can track each student’s weekly progress on two key outcome indicators. We follow a model that evidence building must be an integral component of any innovation, and without it, it becomes impossible to improve the intervention and communicate the benefits of the intervention to key stakeholders.