Comparing Manual Qualitative Coding to Natural Language Processing: A Case Study of Financial Decision-Making
*Names in bold indicate Presenter
Our case study presents a re-analysis of 32 focus group transcripts about financial decision-making. The original analysis was completed by a team of six qualitative researchers who manually coded each statement from a focus group respondent, matched with their responses to closed-ended survey questions. The original analysis identified key themes within four research areas (credit reports and scores, auto purchases, financial rules of thumb, and comparison shopping for financial products). Within each of the four research areas, we identified five to eight topics and associated sub-topics that became the basis for behavior-based consumer segments. The NLP re-analysis will be completed by four researchers with varying levels of programming expertise and use several machine learning techniques (including unsupervised and supervised methods). We will model topics found in the dataset, evaluating results in comparison to the manual coding in terms of NLP’s benefits, challenges, results (including similarity and depth), and tradeoffs between approaches. In addition to comparing the outcomes of the analyses, we will also present information on the level of effort required by each method and skills required of analysts for both analyses. Together the substantive results of the re-analysis and project management knowledge will help policy researchers and evaluators better understand how to select qualitative analysis methods and how to manage projects using both types of analytic methods.