CCE Theses and Dissertations

Date of Award

2025

Document Type

Dissertation

Degree Name

Doctor of Philosophy Cybersecurity Management

Department

College of Computing and Engineering

Advisor

Yair Levy

Committee Member

Gregory Simco

Committee Member

Laurie Dringus

Committee Member

Melissa Carlton

Keywords

adversarial GenAI, DCWF, fundamental competencies, GenAI, human-GenAI teaming, NICE

Abstract

The critical shortage of skilled cybersecurity professionals, with over 750,000 unfilled positions in the United States (U.S.), combined with rising practitioner burnout and the complexity of modern cyber threats, poses significant risks to national security. As Generative Artificial Intelligence (GenAI) emerges as a potential tool to support cybersecurity operations, its role in assisting human analysts with high-demand tasks offers both opportunities and challenges. While GenAI can improve efficiency, it also introduces adversarial risks if manipulated by malicious actors. This study investigated how human-GenAI collaboration can address these challenges, focusing on the fundamental cybersecurity knowledge, skills, and task completion required for effective teamwork, particularly within military contexts where adaptable technologically fluent and savvy "T-SAVVY" soldiers operate in technology-intensive environments.

Using a mixed-methods design, this study engaged 31 novice participants in realistic cybersecurity activities conducted in a commercial Cyber Range. These participants performed tasks independently and with GenAI support (e.g. ChatGPT), to assess the extent to which GenAI enhances cybersecurity competency. This research unfolded in three phases: Phase I involved validating the fundamental cybersecurity competencies and experimental procedures through a Delphi method with 20 Subject Matter Experts (SMEs). In the subsequent phases, novice participants completed tasks in a commercial Cyber Range, designed to measure both competency with and without GenAI support. Furthermore, this research explored the ability to detect adversarial GenAI prompts. Statistical analyses, including Analysis of Variance (ANOVA) and Analysis of Covariance (ANCOVAs), helped evaluate demographic impacts on performance and determine whether GenAI significantly improves cybersecurity outcomes.

Results from ANOVA and ANCOVA analyses confirmed that GenAI support significantly improved mean competency (F=23.523; p-value< 0.001), Knowledge (F=23.992; p-value< 0.001), Skills (F=18.819; p-value< 0.001), and Task completion (F=27.563; p-value< 0.001) scores. However, the time it took to complete the tasks was not significant (F=0.008, p-value=0.930). These improvements reflect notable gains in both efficiency and accuracy when GenAI was used. However, the ability to detect adversarial GenAI responses varied significantly based on participants’ prior experience, emphasizing that while GenAI can enhance novice performance, critical thinking and AI literacy remain essential. Interestingly, while GenAI enhanced task efficiency and accuracy, the difference in adversarial prompt detection was not statistically significant (F=0.873; p-value=0.429), reinforcing the need for human oversight and structured adversarial training. These findings affirm GenAI’s potential as an augmentative—rather than substitutive—tool for cybersecurity, particularly for novice users operating in mission-critical, technology-intensive environments.

The findings are expected to contribute to the Body of Knowledge (BoK) by defining key competencies for GenAI-augmented teams and proposing guidelines for human-GenAI collaboration in the context of cybersecurity. This research aimed to enhance workforce readiness, offering strategies to build a resilient cybersecurity workforce equipped to navigate the complexities of a digitally dependent era.

Download
Share Feedback

Share

COinS