Association for Public Policy Analysis & Management

Identifying specific actions and practices that improve student learning in schools is a perennial, core instructional and policy question. School reformers have promoted a variety of strategies to remedy underperformance in American schools through several decades of whole-school reform efforts from the 1980s and 1990s, to the Title I school accountability sanctions under the No Child Left Behind Act (NCLB) of 2002, and to the Obama administration’s Race to the Top (RttT) and School Improvement Grants (SIG) as part of the American Recovery and Reinvestment Act in 2009. Yet, we still know very little rigorous evidence on what works in which types of schools and communities – the type of knowledge that can be used to develop a theory of action to guide effective school improvement in local contexts (Herman et al., 2008). No matter whether federal or states continue this model of substantial redesign or reconstitution accompanied by administrative accountability under the Obama administration, or the alternative strategy of promoting market and choice under the new Trump administration, the real change and improvement rest on developing effective school practices and personnel behaviors in local contexts. We need to constantly update our understanding of specific organizational and instructional practices that constitute effective schools, as well as develop novel ways of collecting and analyzing data to inform this understanding.

To probe into the processes of schooling and use a subsample of about 400 schools who were under-performing and identified by the State as needed for improvement during 2012-16, we use novel computer-assisted text analysis methods to examine a massive amount of textual data School Comprehensive Planning and Implementation Reports (SCPIRs), which schools are required by the state to report two to four times a year. These reports include 50,000 unique improvement tasks/activities and over 2 million in aggregate. We ask:

1. What are the portfolio of improvement activities in schools?
2. How reliable and valid are these quantitative measures of reform activities?
3. How do reform activities within schools evolve over the course of reform?
4. What are the effective reform activities in what types of schools and communities?

Our work significantly contributes new knowledge of using big volume of text data to support educational reforms. Schools, districts, and states have generated a large amount of textual data and will continue to generate more. Researchers and policymakers have rarely made full use of this large amount of important information to better school and instructional improvement. The current use of the data is much limited by the time-consuming, arduous, and error-prone process of hand-coding. Our efforts in this project have the potential to lead the field to leverage on computer-assisted techniques to efficiently analyze large volume of text data to inform educational policy design and implementation with speed, reliability and validity, on a large scale, and repeatedly over time. Moreover, this study highlights a new technique—extraction, coding, and analyzes of large volume of texts to inform management “mechanism” and “process” beyond traditional survey or interview instruments used in prior policy and management research.