Natural Language Processing for Qualitative Researchers: A Case Study of Financial Decision-Making

Presenter Information

Anna Jefferson, Abt AssociatesFollow

Location

DeSantis Room 1052

Format Type

Plenary

Format Type

Paper

Start Date

16-1-2020 8:45 AM

End Date

16-1-2020 9:05 AM

Abstract

There is tremendous interest in the applications and value of machine learning to qualitative research but few resources to assess how these new techniques compare to established methods. Machine learning methods, namely natural language processing (NLP) have most often been deployed in text analysis projects on scales previously not possible (e.g., scraping social media). What is less well understood is the role NLP may play in more traditional qualitative research, such as interview or focus group studies, and the skills required of qualitative researchers to implement these methods. This paper contributes to an emerging literature on the benefits, requirements, and outcomes of manual qualitative analysis compared to NLP in the social sciences (Guetteman et al 2018; Nelson et al 2017; Laurer et al 2018) and extends it into public policy research.

Our case study reanalyzes 32 focus group transcripts about consumers’ financial decision-making. The original analysis in NVivo identified key themes in four research areas (credit reports, auto financing, rules of thumb, and comparison shopping). These key themes helped us identify behavior-based consumer segments. The reanalysis uses several NLP techniques to model topics found in the dataset. We compare the NLP results to the manual coding in terms of NLP’s benefits, challenges, results (including similarity and depth), and tradeoffs between approaches. We also present the level of effort and skills required of analysts for each analysis. The results will help other qualitative researchers understand how to select qualitative analysis methods in a rapidly evolving landscape for qualitative research.

Keywords

qualitative analysis, machine learning, natural language processing, consumer finance, decision-making

This document is currently not available here.

Share

COinS
 
Jan 16th, 8:45 AM Jan 16th, 9:05 AM

Natural Language Processing for Qualitative Researchers: A Case Study of Financial Decision-Making

DeSantis Room 1052

There is tremendous interest in the applications and value of machine learning to qualitative research but few resources to assess how these new techniques compare to established methods. Machine learning methods, namely natural language processing (NLP) have most often been deployed in text analysis projects on scales previously not possible (e.g., scraping social media). What is less well understood is the role NLP may play in more traditional qualitative research, such as interview or focus group studies, and the skills required of qualitative researchers to implement these methods. This paper contributes to an emerging literature on the benefits, requirements, and outcomes of manual qualitative analysis compared to NLP in the social sciences (Guetteman et al 2018; Nelson et al 2017; Laurer et al 2018) and extends it into public policy research.

Our case study reanalyzes 32 focus group transcripts about consumers’ financial decision-making. The original analysis in NVivo identified key themes in four research areas (credit reports, auto financing, rules of thumb, and comparison shopping). These key themes helped us identify behavior-based consumer segments. The reanalysis uses several NLP techniques to model topics found in the dataset. We compare the NLP results to the manual coding in terms of NLP’s benefits, challenges, results (including similarity and depth), and tradeoffs between approaches. We also present the level of effort and skills required of analysts for each analysis. The results will help other qualitative researchers understand how to select qualitative analysis methods in a rapidly evolving landscape for qualitative research.