CCE Theses and Dissertations

Date of Award

2025

Document Type

Dissertation

Degree Name

Doctor of Philosophy Cybersecurity Management

Department

College of Computing and Engineering

Advisor

Sumitra Mukherjee

Committee Member

Francisco Mitropoulos

Committee Member

Michael Laszlo

Keywords

Cybersecurity, Intrusion Detection System, Machine Learning, Network Anomaly Detection, NSL-KDD Dataset, SMOTE-ENN

Abstract

This dissertation investigates enhanced network anomaly detection using Machine Learning (ML) models. The study addresses two distinct classification problems: binary classification and multiclass classification. In the binary classification task, network traffic data is categorized as either "normal" or "abnormal," where abnormal includes all non-normal traffic. Leveraging the balanced nature of the dataset, this study develops optimized models that achieve consistently high classification performance. Key metrics, including precision, recall, and F1 scores, are used to ensure robust evaluation and reliable detection across all classes.

For multiclass classification, only classes present in both training and test datasets are included to ensure meaningful evaluation. The study focuses on developing models that achieve satisfactory performance for all classes, with a particular emphasis on improving classification metrics for minority classes, which tend to be underrepresented and poorly detected in conventional models.

The research methodology involves training and evaluating various ML models, including Logistic Regression, Random Forest, AdaBoost, LightGBM, and a custom neural network, on the NSL-KDD dataset. Stratified cross-validation and hyperparameter tuning are applied to ensure model robustness and optimal performance. Class balancing techniques, particularly SMOTE-ENN, along with ensemble learning and feature preprocessing, are employed to enhance detection, particularly for minority attack types.

This dissertation's primary contribution is to identify and evaluate high-performing models tailored to both classification scenarios. By refining detection mechanisms, this study presents improved methods to enhance network security through accurate identification of anomalous activities and underrepresented attack types. The findings provide insights into the effectiveness of various ML approaches for anomaly detection, supporting the development of more resilient and efficient intrusion detection systems (IDS).

The study demonstrates that several ensemble-based models, such as Random Forest, XGBoost, and LightGBM, consistently achieve strong performance in both binary and multiclass classification tasks. Additionally, the integration of SMOTE-ENN effectively enhances the detection of minority attack classes. A custom neural network model also outperforms traditional methods in several scenarios, offering a balance between accuracy and execution time. These results underscore the importance of hybrid and balanced approaches in building robust intrusion detection systems.

Download
Share Feedback

Share

COinS