Association for Public Policy Analysis & Management

Roundtable: Interventions for Struggling Readers: When Evidence Meets the Real World

(Education)

In response to No Child Left Behind accountability policies as well as provisions in state and federal legislation over the past 15 years to improve reading instruction and support, districts and states have devoted more resources to serving children struggling to master key reading skills in the early grades. In particular, districts and schools have responded to increased research about effective practices by adopting evidence-based approaches to support struggling readers in elementary school. This increased adoption creates some special challenges for researchers studying the effectiveness of reading support approaches now implemented at scale (rather than in tightly controlled research settings) or who are trying to replicate results of earlier studies. In addition, researchers must now confront a different set of effectiveness research questions, given the expanded use of resources and practices. Rather than merely answering summative questions such as “can a new approach improve reading achievement,” policy researchers must now contend with design questions that address when, for whom, and why reading intervention is effective. In reading in particular, the ability to detect impacts is based on design choices such as: 1) for whom: for students at which level of reading difficulty should reading intervention begin, and how to decide which students are not responding to interventions; 2) when: at what grade should interventions start, and should intervention begin after students demonstrate difficulty with one skill or multiple skills; 3) how should data be used: should intervention be based on multiple observations for increased validity; 4) how much: what should the dosage or intensity of reading support be, and how might that vary by reading level, to ensure serious reading support rather than “quick fixes”? This roundtable discusses papers that address these design questions, and the challenges of replicating results or finding impacts for programs implemented at scale given the variety of design choices in play. The first speaker will discuss results from an RCT that challenge the preventive intent of short-term supplemental tutoring models, and point to the importance of dosage and timing to yield lasting impacts. The second speaker will discuss results from a regression discontinuity design in schools implementing interventions in the "real world," without monitoring from researchers. The results address design choices about whether to wait before assigning students to receive intervention services, and how to distinguish responsive from unresponsive students, particularly among those at the margin of needing reading support. The third speaker will discuss results from 5 RCTs of the same intervention, to illustrate how the changing nature of the counterfactual directly affects researchers’ ability to replicate previously demonstrated effects. The roundtable also includes a discussant with decades of experience with school districts considering reading interventions. She will reflect on how to translate research into practice, particularly when effects for a given reading approach may change in size and significance over time or at scale.