Association for Public Policy Analysis & Management

Panel: New Development of Text-As-Data Methods and Its Applications in Public Policy Research

(Tools of Analysis: Methods, Data, Informatics and Research Design)

New technologies have made large volumes of textual data increasingly available to researchers. Text information from newspapers, governmental documents, social media, and many other sources provide an unprecedented opportunity for social scientists to construct new metrics, look deeper into the process of social phenomena, and answer important or new policy and management questions. However, the high dimensional nature of textual data also poses serious methodological challenges for causal analysis—the most widely used empirical research methods that inform policy design and evaluation. At the same time, though the application of text-as-data methods is seeing a rapid increase, especially in political science research, other areas in social science and public policy studies still have not seen dramatic take-ups. This panel presents both new methodological advance in text analysis and novel applications in both political and educational policy analysis. Our panel aims to provide an opportunity for researchers from a wide range of policy areas to spark conversations on how to use text data in public policy and management research.

The first paper provides a novel solution for analyzing experiments that use high dimensional interventions (e.g., a text). Without coding the high-dimensional intervention into a couple of low-dimensional interventions, the approach proposed in this paper is able to estimate causal effects for the low-dimensional interventions, and construct plausible confidence intervals. The second paper applies computational text reuse method to analyze lawmaking. It first overview text reuse methods. It then shows that the ability to quantify the evolution of bills can shed new light on the legislative process, including how and why bills are changed and eventually become law.

The last two papers represent two applications of text analysis to educational policy research. One paper focuses on school reform. It uses a large volume of text data from school planning and implementation reports generated by underperforming schools in the state of Washington. By measuring reform practices using these text data, this paper contributes to the knowledge of effective school management practices and policies. The other paper looks into classrooms. By using transcription from classroom videos, the author creates metrics that capture effective teacher practices based on the interaction patterns and teacher language. By leveraging an experiment that randomly assigned students to teachers, the author further estimates the causal effects of those practices on student learning outcomes.

This panel features the new direction in computational text analysis, especially as it pertains to public policy research. The presenters include both leading scholars, junior researchers, and practitioners from multiple institutions, disciplines, policy areas, and research perspectives.