Growing concerns about the reproducibility of scientific findings have led to efforts to enhance the transparency and rigor of social science research. Improving the accuracy and precision of theory development and adopting Open Science practices, such as open data, are widely considered effective means to achieve these ends. However, current Open Science practices primarily target quantitative methodologies and tend to ignore qualitative methods, like document analysis, due to perceived insurmountable obstacles. Since document analysis is often crucial for constructing accurate and precise theories, the inability to integrate Open Science practices with qualitative research methods undermines the effectiveness of these solutions to the replication crisis. We advocate for theoretical document analysis for theory construction to adopt Open Science practices and thoroughly discuss the utility, limitations, and implications of using this approach.
Sarah R. Bostrom, Ph.D. (Corresponding Author)is a Research Scientist in the Warfighter Effectiveness Research Center (WERC) at the United States Air Force Academy. She earned her Ph.D. in Criminal Justice from Sam Houston State University. Her research focuses on contextual and social risks of victimization and particularly susceptibility to deceptive communication including cognitive attacks. Much of her cognitive security work is currently under review with preprints available at CrimRxiv and OSF. Some of her recent work appears in Crime & Delinquency,Social Science, and more. Please direct correspondence to Sarah.Bostrom.CTR@afacademy.af.edu
Capt. Gina L. LeValleycurrently serves as a Flight Commander for the 22nd Intelligence Squadron at Fort Meade Maryland. She has a master's degree in Mental Health Counseling from Webster University. Previously, Captain LeValley served as a Senior Instructor and Executive Officer in the Department of Behavioral Sciences and Leadership at the United States Air Force Academy where she taught Introduction to Psychology and Leadership courses.
Katrina Cooleyis an intelligence analyst in a global security operations center. She is working on her Ph.D. in Security Studies at Kansas State University with a focus on how technology has impacted disinformation and propaganda campaigns throughout history. Her recent work appears in IEEE, HICSS, and Frontiers in Psychology.
Richard E. Niemeyer, Ph.D.is an Associate Professor in the Department of Behavioral Sciences & Leadership at the United States Air Force Academy (USAFA) and the Director of the Academy’s Warfighter Effectiveness Research Center (WERC). His recent research investigates how the interactions among cognitive processes, neurobiological mechanisms, and sociotechnological contexts shape personal and social identities, influence criminal and deviant behavior, and alter susceptibility to psychobehavioral manipulation. Additionally, Dr. Niemeyer focuses on refining theoretical and methodological approaches to enhance scientific research’s accuracy, precision, validity, and reproducibility.
Acknowledgements
This work was supported by the Air Force Office of Scientific Research, funding award FA9550-23-1-0453
Bostrom, S. R., LeValley, G. L., Cooley, K., & Niemeyer, R. E. (2025). Open science is coming to theory construction: The case of document analysis. The Qualitative Report, 30(8), 4152-4167. https://doi.org/10.46743/2160-3715/2025.7659
