Natural Language Processing for Qualitative Researchers: A Case Study of Financial Decision-Making
Location
DeSantis Room 1052
Format Type
Plenary
Format Type
Paper
Start Date
16-1-2020 8:45 AM
End Date
16-1-2020 9:05 AM
Abstract
There is tremendous interest in the applications and value of machine learning to qualitative research but few resources to assess how these new techniques compare to established methods. Machine learning methods, namely natural language processing (NLP) have most often been deployed in text analysis projects on scales previously not possible (e.g., scraping social media). What is less well understood is the role NLP may play in more traditional qualitative research, such as interview or focus group studies, and the skills required of qualitative researchers to implement these methods. This paper contributes to an emerging literature on the benefits, requirements, and outcomes of manual qualitative analysis compared to NLP in the social sciences (Guetteman et al 2018; Nelson et al 2017; Laurer et al 2018) and extends it into public policy research.
Our case study reanalyzes 32 focus group transcripts about consumers’ financial decision-making. The original analysis in NVivo identified key themes in four research areas (credit reports, auto financing, rules of thumb, and comparison shopping). These key themes helped us identify behavior-based consumer segments. The reanalysis uses several NLP techniques to model topics found in the dataset. We compare the NLP results to the manual coding in terms of NLP’s benefits, challenges, results (including similarity and depth), and tradeoffs between approaches. We also present the level of effort and skills required of analysts for each analysis. The results will help other qualitative researchers understand how to select qualitative analysis methods in a rapidly evolving landscape for qualitative research.
Keywords
qualitative analysis, machine learning, natural language processing, consumer finance, decision-making
Natural Language Processing for Qualitative Researchers: A Case Study of Financial Decision-Making
DeSantis Room 1052
There is tremendous interest in the applications and value of machine learning to qualitative research but few resources to assess how these new techniques compare to established methods. Machine learning methods, namely natural language processing (NLP) have most often been deployed in text analysis projects on scales previously not possible (e.g., scraping social media). What is less well understood is the role NLP may play in more traditional qualitative research, such as interview or focus group studies, and the skills required of qualitative researchers to implement these methods. This paper contributes to an emerging literature on the benefits, requirements, and outcomes of manual qualitative analysis compared to NLP in the social sciences (Guetteman et al 2018; Nelson et al 2017; Laurer et al 2018) and extends it into public policy research.
Our case study reanalyzes 32 focus group transcripts about consumers’ financial decision-making. The original analysis in NVivo identified key themes in four research areas (credit reports, auto financing, rules of thumb, and comparison shopping). These key themes helped us identify behavior-based consumer segments. The reanalysis uses several NLP techniques to model topics found in the dataset. We compare the NLP results to the manual coding in terms of NLP’s benefits, challenges, results (including similarity and depth), and tradeoffs between approaches. We also present the level of effort and skills required of analysts for each analysis. The results will help other qualitative researchers understand how to select qualitative analysis methods in a rapidly evolving landscape for qualitative research.