Panel: Set Theoretic Methods for Policy Analysis: A Bad Fit?
(Tools of Analysis: Methods, Data, Informatics and Empirical Design)

Thursday, November 6, 2014: 1:00 PM-2:30 PM
Isleta (Convention Center)

*Names in bold indicate Presenter

Panel Organizers:  Sean Tanner, University of California, Berkeley
Panel Chairs:  Jane Mauldon, University of California, Berkeley
Discussants:  Apu Zutshi, Mathematica Policy Research, Inc.


Qca: Claims and Limitations
Sherry Zaks, University of California, Berkeley



Comparing Boolian and Logit Analysis: An Application to Rare Events Data
Kendra Koivu and Chris Butler, University of New Mexico



What Does Qca Add to Policy Analysis?
Sean Tanner, University of California, Berkeley


The use of qualitative and mixed methods research can substantially strengthen causal inference in policy-related social science. Indeed, JPAM has just published a symposium on qualitative research, with a message from the editor underscoring the importance of encouraging mixed-methods research. However, as the qualitative and quantitative research communities look for common ground, it is more than ever important to maintain the high methodological standards that have guided policy research in recent decades. The goal of this panel is to assess a multi-method approach that has been forcefully advocated as distinctively valuable for policy research. Specifically, qualitative comparative analysis (QCA) has been championed as a multi-method approach of special relevance for policy studies. QCA employs a Boolean framework for finding complex, set-theoretic relationships between clusters of explanatory variables and outcomes. It has received considerable attention in sociology and political science, and there is now a major push to introduce it into policy research. Given the new openness to mixed-method research in the discipline, it is imperative to assess the potential of QCA for enhancing causal inference in policy studies. This panel asks whether QCA is in fact useful for policy analysts. Paper 1 introduces QCA’s assumptions and algorithms. It explores the large gap between the claims about the method and the genuine contributions to causal inference. Focusing specifically on the Boolean logical framework, Paper 2 compares the analytic power of QCA versus logistic regression. Using simulations, it shows that QCA’s Boolean results are dependent upon consequential choices made by the researcher, and that these choices have received inadequate attention. Against this backdrop, Paper 3 then focuses on the specific claim that QCA can strengthen causal inference in policy analysis. It examines policy studies from both the QCA and econometric literatures, as well as the results of simulations using QCA on data from a randomized trial. It concludes that QCA’s conceptual framework adds little to policy analysis; the method fails to address the marginal effects that are an absolutely central concern in research on policy impacts; and it is highly vulnerable to both Type I and Type II errors. Taken together, the papers provide an integrated assessment of this new approach to causal inference, with examples drawn from a range of complex, global policy problems, from mitigating climate change to maintaining an educated workforce. The panel participants represent diverse research agendas, methodological orientations, and academic disciplines, from international relations to domestic social policy.