A Generalized and Robust Nonlinear Approach based on Machine Learning for Intrusion Detection

ABSTRACT

Intrusion detection systems (IDS) play a critical role in ensuring the security and integrity of computer networks. There is a constant demand for the development of powerful, novel, and generalized methods for IDS that can accurately detect and classify intrusions. In this study, we aim to evaluate the benefits of linear classifiers (LC) and nonlinear classifiers (NLC) in IDS. We employed ten machine learning (ML) classifiers, consisting of five LC and five NLC. These classifiers underwent cross-validation for performance evaluation, unseen analysis, statistical tests, and power analysis on measuring the minimum sample size. Four hypotheses were formulated and validated on five processed intrusion attack datasets. NLC outperformed LC, with a mean accuracy (ACC)/area-under-the-curve (AUC) increase of 22.26%/20.3% on the WUSTL-EHMS dataset, with improvements of ACC/AUC by 5.5%/2.3% on the UNSW-NB15 dataset. In the unseen analysis, NLC achieved an ACC/AUC increase of 21.9%/21.8% when trained on WUSTL-EHMS and tested on UNSW-NB15. Lastly, when using a mixed dataset of WUSTL-EHMS and UNSW-NB15, NLC demonstrated an ACC/AUC increase of 11.67%/5.5%. The model performed well in cross-validation protocols, and the statistical tests yielded significant p-values. NLC provides generalized and robust solutions to detect intrusion attacks, ensuring the integrity and security of computer networks.

Introduction

Intrusion detection refers to the process of monitoring and analyzing computer networks (Kizza, Kizza, and Wheeler 2013) or systems to identify and respond to unauthorized or malicious activities (Chakraborty 2013). It involves using specialized tools and techniques to detect suspicious behaviors, such as unauthorized access attempts, malware infections (Sapalo Sicato et al. 2019), or unusual network traffic patterns. There are two main types of intrusion detection (Zamani and Movahedi 2013): network-based (Ring et al. 2019) and host-based (Singh and Singh 2014). Network-based monitoring focuses on analyzing network traffic, while host-based monitoring focuses on individual systems or hosts. The benefits of intrusion detection encompass early detection and prevention of security breaches, timely response to incidents, protection of sensitive data and resources, and enhancement of overall system security. Intrusion detection is necessary because cyber threats (Abubakar et al. 2015) and attacks have become increasingly sophisticated and prevalent. Organizations and individuals need intrusion detection systems (IDS) to proactively identify and defend against potential threats (Rubio et al. 2017), safeguard their networks and systems, maintain data confidentiality and integrity, and ensure the smooth functioning of their operations.

In intrusion detection, various methods are utilized to identify and respond to malicious activities (Garcia-Teodoro et al. 2009). Signature-based detection (Kumar and Sangwan 2012) compares observed patterns against known attack signatures, while anomaly-based detection (Jyothsna, Prasad, and Prasad 2011) seeks deviations from normal behavior. Behavior-based detection (Khraisat et al. 2019) analyzes user and system behavior for suspicious activities (Waqas et al. 2022). Packet intrusion (Chen, Chen, and Lin 2010) involves monitoring network packets for signs of intrusions, such as unauthorized access or abnormal traffic (Devarajan and Rao 2021). White list and black list methods (Chauhan et al. 2013; Tiwari et al. 2009) maintain lists of trusted and malicious entities, respectively. In this study, ten machine learning (ML) classifiers are assessed, encompassing both linear classifiers (LC) and nonlinear classifiers (NLC). The analysis incorporates cross-validation, unseen analysis, statistical tests, and power analysis for sample size calculations. Due to the nonlinear nature of the intrusion dataset, it is anticipated that the NLC will outperform the LC. The objective is to evaluate the reliability and robustness of the system through rigorous testing and analysis.

Intrusion detection procedures can be time-consuming due to the volume of network traffic and the need for continuous updates. The emergence of evolving cyber threats, such as man-in-the-middle attacks (Mallik 2019) which exploit unknown vulnerabilities (Rauf et al. 2022), further complicates the detection process. Automation is essential to enable real-time monitoring and response. ML classifiers play a crucial role in analyzing network data, identifying anomalies, and detecting various types of attacks (Zhang, Hu, and Feng 2010). ML classifiers have the ability to adapt and learn, making them highly valuable in handling evolving intrusion attempts. By automating intrusion detection using ML classifiers (Pai and Adesh 2021), security, accuracy, and response time can be enhanced, effectively mitigating potential threats in modern networks.

In our study, we focused on analyzing the performance of both LC and NLC in the context of IDS. The results of our study supported the hypothesis that the intrusion dataset exhibits nonlinear characteristics, indicating the superiority of NLC over LC. To validate the reliability of the system, we conducted an unseen analysis, comparing the performance of LC and NLC on previously unseen intrusion patterns. The results confirmed the effectiveness of NLC in accurately detecting these unseen intrusion patterns. Additionally, we demonstrated the robustness of the system through statical tests, with a highly significant p-value below 0.0001. We emphasized the importance of sample size calculation for power analysis, emphasizing the need for larger sample sizes to ensure statistically significant results and enhance the reliability of our findings. By utilizing both LC and NLC, our study aims to achieve high detection accuracy and robustness against a wide range of cyber threats. We believe that our interdisciplinary approach contributes to the advancement of cybersecurity by providing effective solutions for network protection and addressing the evolving nature of cyber threats.

The paper follows a structured approach, where it begins with a literature overview in Section 2, while Section 3 presents the dataset description, data preprocessing techniques, class balancing methods, and cross-validation. Section 4 includes the experimental discussions, focusing on the performance of LC and NLC and presenting the hypothesis. The overall results of the LC and NLC on different dataset combinations are showcased in Section 5. Section 6 covers performance evaluations, validation methods, and statistical tests, and includes the presentation of the ROC curve. Section 7 consists of overall discussions, including key findings, benchmarking against previous studies, and strengths, weaknesses, and future research directions. Finally, Section 8 concludes the paper.

Literature Overview

Traditional intrusion detection methods (Viegas, Santin, and Abreu 2021) commonly rely on rule-based and signature-based approaches (Yang et al. 2013). Rule-based methods involve defining specific patterns or rules to detect known attacks, while signature-based methods utilize predefined signatures or patterns of known attacks for detection. However, these methods have limitations. They depend on static rules or signatures, making them less effective against novel or unknown attacks. Moreover, they may produce false positives or negatives, which hinder their accuracy and scalability (Naiping and Genyuan 2010).

In recent years, several research works have focused on intrusion detection using ML techniques, as presented in Table 1. Kilincer et al. (Kilincer, Ertam, and Sengur 2021) reviewed decision tree (DT), k-nearest neighbors (KNN), and support vector machine (SVM) models on datasets such as CSE-CIC-IDS-2018 (Leevy and Khoshgoftaar 2020), ISCX 2012 (Yassin et al. 2013), NSL-KDD (Dhanabal and Shantharajah 2015), CIDDS-001 (Ring et al. 2017), and UNSW-NB15 (Moustafa and Slay 2016). Although the models achieved high accuracy, precision, and recall values of 0.99, the absence of an unseen analysis to evaluate the generalizability of the results suggests potential over fitting. Sarker et al. (2020), used a DT-based model, to classify normal and anomaly samples, achieving an accuracy, precision, recall, and F-score of 0.98. However, the study relied on traditional feature ranking methods (Chang and Lin 2008), indicating a limitation in the design of the IDS.

Table 1. Overview of existing studies.

Year	Author	Methodology	Advantages/Disadvantages	Evaluation
2019	Bindra and Sood (2019)	Utilizing standard machine learning models for detecting Distributed Denial of Service (DDoS) attacks	Compared results using a variety of machine learning models. The models employed were very basic, and no deep learning model or feature extraction was performed.	96.2% Accuracy (CIC IDS2017)
2020	Sarker et al. (2020)	Built a tree-based generalized intrusion detection model using the ranking of security features.	Selected important security features with minimal computational complexity for the system. No deep learning-based models were used.	98% Accuracy
2023	Zhang et al. (2023)	Developed an intrusion detection algorithm that optimizes model parameters through many-objective optimization with double evolutionary selections.	Achieved multi-knowledge fusion and enhanced performance through internal and external population knowledge.	99.28% Accuracy
2023	Tu et al. (2023)	Implemented a Pseudo-Siamese stacked autoencoders to combine unsupervised and supervised learning for extracting semantic features from network traffic.	Observed significant performance improvement was due to better extraction of semantic features. Training and inference time should be provided to account for additional computational overhead.	90.05% Accuracy (KDDTEST+)
2023	Friha et al. (2023)	Decentralized and differentially private federated learning-based intrusion detection mechanism for industrial IoT ecosystem	Enhanced classification beyond the standard federated learning process. Inherent trade-off between privacy and performance remains a standard challenge.	94.37% Accuracy (Centralized learning) 93.91% Accuracy (Federated)
2024	Mármol Campos et al. (2024)	Developed a federated learning architecture for intelligent transportation systems using a collaborative learning approach and employed a federated Multi-layer Perceptron model for misbehavior detection.	Maintained a balance between the performance of the federated model and the training delay. There is a need to increase the effectiveness of classification	93% Accuracy (MulticlassClassification)

Zhang et al. (2023) developed a Many-Objective Optimization-Based intrusion detection system for securing in-vehicle networks, using an algorithm that adopts double evolutionary selections to optimize the detection model parameters. Bindra and Sood (2019) focused specifically on DDoS attack detection (ParwANI et al. 2015). The study employed eight classifiers, including LR, KNN, Gaussian Naive Bayes (GNB), Random Forest (RF), Linear SVM, KNN, RF, and Linear Discriminant Analysis (LDA). Tu et al. (2023) proposed an intrusion detection mechanism deployed in a fog environment using pseudo-siamese stacked autoencoders. To extract deep semantic features, unsupervised training was performed on the stacked autoencoders, followed by supervised learning with labels.

Federated learning based schemes have also been proposed for IDS systems. Mármol Campos et al. (2024) designed a federated learning architecture for intelligent transportation systems using collaborative learning approach. Friha et al. (2023). also proposed a similar approach using differentially private federated learning-based intrusion detection system for industrial IoT ecosystem. The system comprised of three components, a key exchange protocol, a differentially private gradient exchange scheme, and a decentralized federated learning approach.

Previous research in intrusion detection has limitations, including overfitting, lack of cross-validation, and inadequate methods for handling imbalanced class issues. In contrast, this proposed paper addresses these flaws by implementing rigorous quality control measures, validating hypotheses through unseen analysis, and conducting statistical tests. Emphasizing generalization and power analysis, the study calculates optimal sample sizes for training and testing models. By considering these factors, the proposed work aims to establish a reliable and robust IDS, overcoming the limitations observed in previous studies. Experimental and theoretical approaches are employed for validation, ensuring the credibility of the proposed hypotheses.

Materials and Methods

The research included the use of statistical ML models for the categorization to gain a deeper knowledge of the link between datasets and their associated features. The first step was to collect the dataset that would be used for classification. Next, it was important to perform data pre-processing tasks, the problem of missing values, scaling the data, over sampling the minority class, and using label encoder attributes encoding. Additionally, the performance measures are determined that would be used to access the models by computing features from the dataset. Acronyms and symbols used throughout the paper have also been declared in Tables 2 and 3 respectively.

Table 2. Acronyms used in the manuscript.

SN	Abb*	Definition	SN	Abb*	Definition
1	ANOVA	Analysis of Variance	13	MCC	Matthew’s Correlation Coefficient
2	AUC	Area-under-the-curve	14	ML	Machine Learning
3	DL	Deep Learning	15	NB	Naive Bayes
4	DT	Decision Tree	16	NLC	Nonlinear Classifier
5	EHMS	Enhanced Healthcare Monitoring System	17	PERPT	Perceptron
6	IDS	Intrusion detection systems	18	RF	Random Forest
7	GBM	Gradient Boosting Machine	19	ROC	Receiver Operating Characteristic
8	KNN	K-Nearest Neighbors	20	RR	Ridge Regression
9	LC	Linear Classifier	21	SMOTE	Synthetic Minority Over-sampling Technique
10	LDA	Linear Discriminant Analysis	22	SVM	Support Vector Machine
11	LR	Logistic Regression	23	XGB	Extreme Gradient Boosting
12	MAC	Media Access Control address

Abb* = Abbreviation.

Table 3. Symbols used in the manuscript.

SN	Symbols	Explanation
1	$\mathit{\eta }$	Accuracy
2	$\mathrm{R}$	Recall
3	$\mathrm{P}$	Precision
4	F	F1-Score
5	$\mathit{S}$	Specificity
6	${\mathrm{n}}_{\mathrm{s}}$	Minimum Sample size
7	$\tilde{}\mathrm{p}$	The estimated proportion of the characteristic in the population
8	MoE	Margin of Error

Dataset Description

We utilized five carefully selected datasets for intrusion detection, providing diversity in inspection and enabling validation of unseen scenarios. By utilizing these datasets, each with its unique network configurations, traffic patterns, and types of attacks, we aimed to capture a comprehensive representation of real-world intrusions. This approach helped us avoid biased results and improve the effectiveness of our intrusion detection algorithms. Moreover, using multiple datasets ensured the reliability of our system across diverse network environments.

Dataset 1: WUST-EHMS

The WUSTL-EHMS-2020 dataset (Tauqeer et al. 2022) is a unique collection derived from a real-time Enhanced Healthcare Monitoring System (EHMS) spoofing (Kim et al. 2020). The EHMS testbed includes medical sensors, a gateway, network components, and a control system with visualization. In the dataset, data flow starts from patient sensors, passes through the gateway, and is then sent to the server. However, this transmission is vulnerable to interception and modification attacks. The dataset focuses on man-in-the-middle attacks, specifically spoofing (Kim et al. 2020) and data injection attacks (Sayghe et al. 2020). Spoofing compromises patient data confidentiality, while data injection attacks compromise data integrity.

The IDS captures network flow traffic and detects abnormalities in both network flow and patient biometric data. It contains 44 features, with the Source MAC address (Tao, Li, and Sampalli 2008) used for labeling. Samples with the attacker’s laptop MAC addresses are labeled as 1, while the rest are labeled as 0. The dataset comprises 16,318 samples, including 14,272 normal samples and 2,046 attack samples.

Dataset 2: UNSW-NB15

The UNSW-NB15 dataset (Souhail 2019) is a comprehensive collection of network packets generated using the IXIA Perfect Storm tool in the Australian Centre for Cyber Security’s (ACCS) Cyber Range Lab. It offers a blend of real-world normal activities and synthetic contemporary attack behaviors. The dataset encompasses nine types of attacks, including Fuzzers, Backdoors, Analysis, Exploits, DoS, Generic, Shellcode, Reconnaissance, and Worms (Rashid 2020). The dataset was generated using tools like Argus and Bro-IDS, employing twelve algorithms that produced 49 features for each network packet. Class labels were assigned based on whether a sample represented an attack or normal activity, with attacks labeled as 1 and normal samples as 0. The dataset comprises 82,334 samples, including 37,000 normal samples and 45,332 attack samples.

Dataset 3: Unseen 1 – Train on WUSTL-EHMS and Test on UNSW-NB15

For our first unseen dataset analysis, we combined two datasets to train and test our IDS. The training dataset comprised a fusion of five datasets from WUSTL-EHMS, each containing 28,544 samples. The testing dataset encompassed ten datasets from UNSW-NB15, with each dataset containing 8,233 samples. Our primary goal was to enhance the performance and generalization capability of our IDS by leveraging multiple datasets. Additionally, we conducted an analysis of this unseen data to evaluate the system’s effectiveness in detecting and classifying previously unseen intrusion patterns. This comprehensive approach allowed us to assess the system’s reliability and robustness in real-world scenarios, where emerging types of attacks may pose a threat. Through the amalgamation of these datasets and thorough analysis, our research contributes to advancing intrusion detection techniques and fortifying computer network security against evolving cyber threats.

Dataset 4: Unseen 2 – Train on UNSW-NB15 and Test on WUSTL-EHMS

For our second unseen dataset analysis, we employed a combined dataset comprising UNSW-NB15 and WUSTL-EHMS to train and test our IDS. The training dataset encompassed five datasets from UNSW-NB15, totaling 90,664 samples and providing a substantial volume of network traffic data. On the other hand, the testing dataset consisted of ten datasets from WUSTL-EHMS, with each dataset containing 1,631 samples. By integrating both datasets, our aim was to capture a diverse range of network behaviors and intrusion patterns, thereby enhancing the system’s capability to detect and classify various types of attacks. Moreover, we conducted an analysis of previously unseen intrusion scenarios to assess the system’s performance, ensuring its robustness and reliability.

Dataset 5: Mix-Matching Using WUST-EHMS and UNSW-NB15 Datasets

We adopted a mix-matching approach to creating the last dataset for our unseen analysis. This dataset was formed by combining Dataset WUSTL-EHMS and Dataset UNSW-NB15 for training and testing purposes. By combining and mixing the data from multiple datasets, our intention was to introduce greater diversity and variability in the training and testing samples. This approach enables us to evaluate the performance and generalization capabilities of our IDS on unseen instances that may exhibit different characteristics or patterns compared to the original datasets. By utilizing this mixed dataset, we can effectively assess the system’s ability to detect and classify intrusions in real-world scenarios involving various attack types and network behaviors.

Quality Control

Quality control is crucial in IDS when using datasets like WUSTL-EHMS and UNSW-NB15, requiring careful validation through data pre-processing, including cleaning, normalization, and handling missing values. Feature selection eliminates irrelevant features, reducing complexity and noise, and improving system performance.

Data preprocessing (Davis and Clark 2011) plays a crucial role in working with the WUSTL-EHMS and UNSW-NB15 datasets, ensuring data quality and preparing it for analysis. For both datasets, thorough examination and processing of each attribute are conducted. Data cleaning (Alrowaily, Alenezi, and Lu 2019) is performed to handle missing values appropriately. Linear interpolation is applied to replace the missing values, estimating them based on a linear relationship between available data points. By employing linear interpolation on the four instances with missing values, the dataset is successfully completed, ensuring data integrity for further analysis. Unnecessary attributes that have minimal contribution to the analysis are eliminated through data reduction, reducing computational complexity. Data transformation techniques (Revathi and Malathi 2013), such as standard scaling, are implemented to normalize the data and bring it to a standard scale, facilitating comparability among different attributes. This is achieved by subtracting the minimum value and dividing by the range (Johri et al. 2022; Konstantonis et al. 2022). By standardizing the features using this method, the values are reduced, which expedites the training process for both ML models. To enable the application of ML algorithms, label encoding is utilized to convert categorical labels into numerical representations.

To address the class imbalance in the WUSTL-EHMS and UNSW-NB15 datasets, we applied data augmentation methods, specifically the Synthetic Minority Over-sampling Technique (SMOTE) (Tan et al. 2019). SMOTE generates synthetic samples for the minority class by interpolating between existing neighboring instances. It is particularly useful for nonlinear datasets with skewed instance distributions, as it helps balance the classes and prevent biased model training. By introducing synthetic samples, SMOTE increases the representation of the minority class, enhancing the classifier’s ability to accurately detect intrusions. This technique is crucial in intrusion detection, as it ensures that the model is not biased toward the majority class, enabling improved intrusion detection and maintaining a high level of system security.

Global Architecture

The computerized paradigm employed in this study is illustrated in Figure 1. Pre-processing steps for the intrusion detection data involve interpolation, scaling, class balancing, and class encoding (Le, Oktian, and Kim 2022). The attributes within the data serve as feature inputs for the prediction module, which employs both LC and NLC to forecast the labels. In the performance evaluation stage, ROC curves are generated using ground truth tables to assess the predicted labels, and statistical tests are conducted to evaluate reliability and stability.

Figure 1. The overall architecture of our computerized system. ML: Machine Learning; DT: Decision Tree; LDA: Linear Discriminant Analysis; LR: Logistic Regression; RR: Ridge Regression; NB: Naive Bayes; PERPT: Perceptron; RF: Random Forest; GBM: Gradient Boosting Machine; KNN: K-Nearest Neighbours; XGB: Extreme Gradient Boosting.

ML Classifiers

In this ML-based intrusion detection study, we employed a combination of both LC and NLC to achieve accurate and robust results. LC, including LR (Belavagi and Muniyal 2016), Ridge Regression (RR) (Ao 2021), Naive Bayes (NB) (Amor, Benferhat, and Elouedi 2004), Perceptron (PERPT) (Rahman et al. 2020), and LDA (Ibrahimi and Ouaddane 2017), are known for their simplicity and interpretability, making them suitable for detecting patterns in data with linear relationships. On the other hand, NLC such as DT (Bhavani, Rao, and Reddy 2019), RF (Bhavani, Rao, and Reddy 2019), Gradient Boosting Machine (GBM) (Khafajeh 2020), K-Nearest Neighbors (KNN) (Liu et al. 2022), and Extreme Gradient Boosting (XGB) (Jiang et al. 2020) excel at capturing complex patterns and relationships in the data, making them well-suited for datasets with nonlinear characteristics. Given that the WUSTL-EHMS and UNSW-NB15 datasets used in this study are likely to exhibit both linear and nonlinear patterns, it was necessary to combine both types of classifiers to effectively detect intrusions and improve overall detection performance. By leveraging the strengths of both LC and NLC, this approach aims to achieve higher accuracy and better performance in detecting various types of intrusions present in the datasets.

Performance Metrics

The proposed research model is evaluated for the classification of attack and normal samples in intrusion detection (Almseidin et al. 2017). The evaluation involves considering parameters such as true positive (TP), false positive (FP), true negative (TN), and false negative (FN). A TP occurs when an intrusion attempt is correctly detected, while a TN refers to correctly identifying a benign attempt as not an intrusion. On the other hand, an FP occurs when a benign event is mistakenly detected as an intrusion, and an FN is when an intrusion attempt is incorrectly identified as benign. Based on these parameters, the following performance evaluation (PE) metrics are derived: (i) Accuracy (η): It measures the model’s overall prediction’s correctness. (ii) Recall ($\mathrm{R}$): Also known as sensitivity, it measures the model’s ability to correctly classify positive instances. (iii) Precision ($\mathrm{P}$): It quantifies the accuracy of positive predictions made by a model. (iv) F1 Score (F): The F1 score is a combined metric that is the harmonic mean of precision and recall, providing a balance between the two. (v) Specificity (S): It measures the model’s ability to correctly identify negative instances. (vi) Matthew’s Correlation Coefficient (MCC): MCC is a correlation coefficient that takes into account all four possible outcomes (TP, TN, FP, FN). It assesses the quality of binary classifications.

(1) $\mathit{\eta }=\frac{(\mathrm{TP}+\mathrm{TN})}{(\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN})}$(1)

(2) $\mathrm{R}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$(2)

(3) $\mathrm{P}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$(3)

(4) $\mathit{F}=2\ast \frac{\left(\mathrm{P}\ast \mathrm{R}\right)}{\left(\mathrm{P}+\mathrm{R}\right)}$(4)

(5) $\mathit{S}=\frac{\mathrm{TN}}{\left(\mathrm{TN}+\mathrm{FP}\right)}$(5)

(6) $\mathit{MCC}=\frac{(\mathrm{TP}\ast \mathrm{TN}-\mathrm{FP}\ast \mathrm{FN})}{\sqrt{(\left(\mathrm{TP}+\mathrm{FP}\right)\ast \left(\mathrm{TP}+\mathrm{FN}\right)\ast \left(\mathrm{TN}+\mathrm{FP}\right)\ast \left(\mathrm{TN}+\mathrm{FN}\right))}}$(6)

Experimental Protocols

To validate our hypotheses for intrusion detection, we collected two datasets: the WUSTLH-EHMS and the UNSW-NB15 dataset. We used these datasets for training as well as testing, specifically aimed at validating our first hypothesis. Additionally, we conducted unseen analysis by training with the WUSTL-EHMS dataset and testing with UNSW-NB15 dataset, as well as training with UNSW-NB15 dataset and testing with the WUSTL-EHMS dataset, to further validate our second hypothesis. We used the K10 protocol as a standard for our evaluation methodology. In this approach, the entire dataset is divided into 10 folds, with one fold randomly selected for testing while the remaining nine folds are used for training the data sets. This process ensures that each data point is utilized for both training and testing, contributing to a more comprehensive assessment of the algorithm’s performance (Jamthikar et al. 2021; Srivastava, Singh, and Suri 2019). Since the process of training is conducted 10 times, the process takes all 10 combinations for evaluation of the performance evaluation (Biswas et al. 2018; Kuppili et al. 2017; Srivastava, Singh, and Suri 2020; Suri and Rangayyan 2006). Additionally, we implemented nested cross-validation to manage parameter tuning independently from the outer loop of the K10 protocol, preventing over fitting and enhancing the reliability of the performance metrics (Maniruzzaman et al. 2019; Teji et al. 2022), which gives the robust performance. Note that such applications are well suited for different healthcare applications and are very powerful leading to generalization (Araki et al. 2016; Shrivastava et al. 2017).

In further analysis, we utilized a combination of mix-match datasets for further unseen analysis. The supervised ML algorithms were categorized into LC and NLC. The LC category consisted of five models: LR (Belavagi and Muniyal 2016), RR (Ao 2021), NB (Amor, Benferhat, and Elouedi 2004), PERPT (Rahman et al. 2020), and LDA (Ibrahimi and Ouaddane 2017). The NLC category included five models: DT (Bhavani, Rao, and Reddy 2019), RF (Bhavani, Rao, and Reddy 2019), GBM (Khafajeh 2020), KNN (Liu et al. 2022), and XGB (Jiang et al. 2020). During the evaluation, various metrics were used: accuracy, sensitivity (recall), precision, F1 score, specificity, MCC, and AUC.

Experiment 1: NLC Vs. LC

The experiment aims to compare the effectiveness of LC and NLC in intrusion detection. The study involved training and testing ten classifiers, including five LC and five NLC, on two intrusion datasets (WUSTL-EHMS and UNSW-NB15). The dataset for intrusion detection encompasses various elements that interact nonlinearly, including user behaviors, network traffic patterns, and attack characteristics. The performance of each classifier was evaluated using the K10 protocol setup. The results were averaged to obtain the final performance metrics, which were used to compare the performance of the LC vs. NLC.

Experiment 2: Unseen Analysis of NCL Vs. LC

The purpose of this experiment was to validate the hypothesis that NLC outperform LC in intrusion detection, specifically under unseen analysis. The study involved training and testing ten classifiers, consisting of five LC and five NLC, on two unseen intrusion datasets. For the first scenario, the classifiers were trained on WUSTL-EHMS dataset and tested on UNSW-NB15 dataset. In the second scenario, the classifiers were trained on the UNSW-NB15 dataset and tested on the WUSTL-EHMS dataset. The performance of each classifier was evaluated using the K10 protocol setup, and the results were averaged to obtain the final performance metrics. These metrics were then used to compare the performance of LC versus NLC.

Experiment 3: NLC Vs. LC on Mix-Matching Dataset

This experiment aimed to further solidify the hypothesis that NLC outperform LC in intrusion detection, particularly under unseen analysis. To achieve this, an additional analysis was conducted using a combination of mix-match datasets. In this analysis, the fifth dataset was utilized to train and test ten classifiers, consisting of five LC and five NLC. The performance metrics obtained from these classifiers were averaged to compare the performance of LC and NLC.

Experiment 4: Effect of Training Data Size on Performance of LC and NLC

In order to validate the stability of the system, we employed four cross-validation protocols: K2, K4, K5, and K10. These protocols allowed us to modify the training data size for each ML model and observe the resulting performance drop that occurred as a result of reducing the training size. The Mix-Matching Dataset-5 was used with these partition protocols, and the results were averaged to illustrate the impact of data size on our models.

Results

We conducted experiments involving five combinations of datasets to analyze the performance of our models effectively and obtain reliable findings. The first combination involved training and testing our models on the WUSTL-EHMS dataset. The second combination involved training and testing on the UNSW-NB15 dataset. In the third combination, we trained on the WUSTL-EHMS dataset and tested on the UNSW-NB15 dataset. Conversely, in the fourth combination, we trained on the UNSW-NB15 dataset and tested on the WUSTL-EHMS dataset. Lastly, we created a fifth dataset by merging data from both the WUSTL-EHMS and UNSW-NB15 into combined training and testing datasets.

To perform our experiments, we utilized the Python sklearn module for ML models. The experiments were conducted on Google Colab. We employed the cross-validation technique with partition protocols K2, K4, K5, and K10. This process allowed us to assess the performance of our models comprehensively. By analyzing the models’ performance using these protocols, we obtained reliable findings for our study. The simulation environment parameters are presented in Table 4.

Table 4. Parameters used in simulation environment and hyperparameters used for in ML classifiers.

Parameter	Description
Features Used	[“DIntPktAct,” “Dport,” “DstBytes,” “DstGap,” “DstJitter,” “DstLoad,” “Dur,” “Loss,” “Packet_num,” “Rate,” “SIntPktAct,” “Sport,” “SrcBytes,” “SrcGap,” “SrcJitter,” “SrcLoad,” “Trans,” “pDstLoss,” “pSrcLoss”]
Target	“Label”
Platform Used	Google Colab, Wireshark, Enhanced Healthcare Monitoring System
Libraries	Numpy, Pandas, Scikit-learn, Matplotlib
Classifier Used	Hyperparameters
Logistic Regression	max_iter: 5000, penalty: “l2”
Ridge Classifier	alpha: 1.0
Gaussian Naive Bayes	None
SGD Classifier	loss: “perceptron,” penalty: “elasticnet,” alpha: 0.0001, max_iter: 1000
Linear Discriminant Analysis	solver: “eigen,” shrinkage: “auto”
Decision Tree Classifier	max_depth: 500, criterion: “entropy,” min_samples_split: 2, min_samples_leaf: 1, ccp_alpha: 0.01, class_weight: “balanced”
Random Forest Classifier	n_estimators: 100, random_state: 42, class_weight: “balanced,” ccp_alpha: 0.01, min_samples_split: 2, min_samples_leaf: 1
Gradient Boosting Classifier	n_estimators: 100, random_state: 42
K-Nearest Neighbors Classifier	n_neighbors: 5
XGBoost Classifier	n_estimators: 100, random_state: 42

NLC vs. LC on Seen Dataset

To validate the hypothesis that NLC exhibits superior performance compared to LC in the context of intrusion detection, an experimental evaluation was conducted. The analysis involved comparing the performance of LC and NLC using two datasets: the WUSTL-EHMS dataset, which contains network flow metrics presented in Table 5, and the UNSW-NB15 dataset, consisting of network flow measurements presented in Table 6.

Table 5. ML model performance on WUSTL-EHMS dataset with K10 protocol.

Classifiers	Accuracy	Recall	Precision	F1 Score	Specificity	MCC	AUC
Performance of Five LC
LR	75.69	0.62	0.91	0.76	0.9	0.65	0.804
RR	72	0.68	0.79	0.72	0.81	0.64	0.785
NB	72.35	0.59	0.9	0.76	0.92	0.69	0.776
PERPT	69.55	0.69	0.88	0.79	0.68	0.67	0.743
LDA	69.23	0.64	0.86	0.71	0.84	0.72	0.768
Mean of LC	71.76	0.64	0.87	0.75	0.83	0.67	0.775
Performance of Five NLC
DT	93.01	0.92	0.95	0.94	0.95	0.92	0.976
RF	94.68	0.96	0.96	0.96	0.96	0.95	0.965
GBM	99.32	0.99	1	0.99	0.96	0.98	0.994
KNN	85.14	0.84	0.93	0.88	0.93	0.94	0.967
XGB	97.95	0.94	0.98	0.96	0.99	0.95	0.989
Mean of NLC	94.02	0.93	0.96	0.95	0.96	0.95	0.978
Absolute Increase (%)	22.26	29	9	20	13	28	20.3

Table 6. ML model performance on UNSW-NB15 dataset with K10 protocol.

Classifiers	Accuracy	Recall	Precision	F1 Score	Specificity	MCC	AUC
Performance of Five LC
LR	94.33	0.99	0.99	0.99	0.99	0.99	0.994
RR	94.98	0.95	0.98	0.96	0.98	0.91	0.963
NB	97.41	0.93	0.99	0.97	0.98	0.95	0.995
PERPT	88.92	0.8	0.91	0.84	0.99	0.82	0.93
LDA	94.95	0.91	0.93	0.92	0.95	0.93	0.987
Mean of LC	94.12	0.92	0.96	0.94	0.98	0.92	0.974
Performance of Five NLC
DT	99.98	0.98	0.99	0.99	0.98	0.99	0.996
RF	99.65	1	1	1	0.99	0.99	0.998
GBM	99.54	0.99	0.99	0.99	0.97	1	0.999
KNN	98.96	0.97	0.99	0.98	0.97	0.97	0.991
XGB	99.99	0.99	1	0.99	0.98	0.99	0.999
Mean of NLC	99.62	0.99	0.99	0.99	0.98	0.99	0.997
Absolute Increase (%)	5.5	7	3	5	0	7	2.3

The evaluation was conducted using performance metrics including accuracy, sensitivity (recall), precision, F1 score, specificity, MCC, and AUC to provide a comprehensive understanding of the classifiers’ effectiveness. The results, presented in Tables 5 and 6, clearly demonstrate that NLC outperformed LC. The mean accuracy difference between the NLC and LC on WUSTL-EHMS dataset across all classifiers was 22.26%, while the mean AUC difference was 20.3%. Additionally, on the UNSW-NB15 dataset, the mean accuracy difference between the NLC and LC across all classifiers was 5.5%, while the mean AUC difference was 2.3%. These findings validate our hypothesis that NLC performs better due to the nonlinear nature of the intrusion dataset. NLC excel at capturing complex nonlinear relationships between input features and output labels. They achieve this by recursively splitting the data into smaller subsets, enabling them to make accurate predictions. This capability is crucial when dealing with intrusion datasets that exhibit nonlinear characteristics.

Unseen Analysis of NLC vs. LC

In order to validate the hypothesis that NLC outperforms LC and ensure the reliability of the IDS, two unseen analyses were conducted. The first part of the analysis involved training the classifiers on the WUSTL-EHMS dataset and testing them on the UNSW-NB15 dataset, with the results presented in Table 7. The second part of the analysis involved training the classifiers on the UNSW-NB15 dataset for training and the WUSTL-EHMS dataset for testing, with the results presented in Table 8.

Table 7. Model performance on unseen analysis-1 train on WUSTL-EHMS and test on UNSW-NB15 using K-10 protocols.

Classifiers	Accuracy	Recall	Precision	F1 Score	Specificity	MCC	AUC
Performance of Five LC
LR	79.36	0.55	0.87	0.71	0.59	0.41	0.828
RR	75.54	0.71	0.85	0.79	0.79	0.5	0.794
NB	58.25	0.77	0.78	0.78	0.65	0.63	0.641
PERPT	71.48	0.85	0.82	0.81	0.76	0.45	0.708
LDA	73.63	0.53	0.6	0.58	0.62	0.65	0.756
Mean of LC	71.65	0.68	0.78	0.73	0.68	0.53	0.745
Performance of Five NLC
DT	93.55	0.54	0.81	0.66	0.63	0.88	0.987
RF	91.69	0.66	0.8	0.75	0.75	0.64	0.942
GBM	84.98	0.8	0.83	0.8	0.9	0.81	0.911
KNN	100	0.92	0.94	0.93	0.91	0.68	0.99
XGB	97.54	0.86	0.89	0.88	0.86	0.65	0.984
Mean of NLC	93.55	0.76	0.85	0.8	0.81	0.73	0.963
Absolute Increase (%)	21.9	8	7	7	13	20	21.8

Table 8. Model performance on unseen analysis-2 train on UNSW-NB15 and test on WUSTL-EHMS with K10 protocol.

Classifiers	Accuracy	Recall	Precision	F1 Score	Specificity	MCC	AUC
Performance of Five LC
LR	82.25	0.81	0.54	0.7	0.84	0.9	0.884
RR	70.12	0.69	0.75	0.72	0.85	0.84	0.743
NB	78.43	0.48	0.67	0.58	0.39	0.56	0.867
PERPT	75.96	0.75	0.79	0.75	0.92	0.76	0.823
LDA	72.65	0.78	0.64	0.72	0.83	0.75	0.799
Mean of LC	75.88	0.7	0.68	0.69	0.77	0.76	0.823
Performance of Five NLC
DT	99.98	0.99	0.99	0.99	0.84	0.99	0.999
RF	99.21	0.99	0.75	0.87	0.91	0.96	0.999
GBM	99.86	0.98	0.46	0.71	0.96	0.81	0.998
KNN	98.99	0.95	0.82	0.89	0.89	0.63	0.989
XGB	99.75	0.94	0.54	0.74	0.82	0.74	0.989
Mean of NLC	99.56	0.97	0.71	0.84	0.88	0.83	0.995
Absolute Increase (%)	23.68	27	3	15	11	7	17.2

The evaluation was conducted using performance metrics including accuracy, sensitivity (recall), precision, F1 score, specificity, MCC, and AUC to provide a comprehensive understanding of the classifiers’ effectiveness. The results, presented in Tables 7 and 8, clearly demonstrate that NLC outperformed LC. The mean accuracy difference between the NLC and LC in Unseen Analysis-1 across all classifiers was 21.9%, while the mean AUC difference was 21.8%. Additionally, in Unseen Analysis-2, the mean accuracy difference between the NLC and LC across all classifiers was 23.68%, while the mean AUC difference was 17.2%. These findings validate our hypothesis that NLC generalizes better over new dataset due to the nonlinear nature of the intrusion. NLC are robust enough in detecting previously unseen intrusion patterns. The results of this analysis provided valuable insights into the comparative performance of LC and NLC under unseen conditions, contributing to the enhancement of IDS and the overall reliability of the system.

NLC vs. LC on Mix-Matching Dataset

To further validate the hypothesis that NLC outperforms LC and ensure the reliability of the system, an additional analysis was conducted using a combination of mix-match datasets. The fifth dataset consisted of merging data from the WUSTL-EHMS and the UNSW-NB15 dataset, into separate training and testing data using the combined data. The objective of using this mix-match dataset was to evaluate the performance presented in Table 9, comparing NLC to LC under unseen analysis.

Table 9. Model performance on mix-matching dataset with K10 protocol.

Classifiers	Accuracy	Recall	Precision	F1 Score	Specificity	MCC	AUC
Performance of Five LC
LR	92.14	0.91	0.9	0.91	0.92	0.9	0.976
RR	90.01	0.88	0.92	0.9	0.83	0.78	0.972
NB	69.97	0.84	0.65	0.73	0.74	0.65	0.867
PERPT	87.34	0.97	0.91	0.93	0.91	0.82	0.895
LDA	89.25	0.91	0.89	0.9	0.85	0.79	0.911
Mean of LC	85.74	0.9	0.85	0.87	0.85	0.79	0.924
Performance of Five NLC
DT	98.07	0.98	0.99	0.98	0.99	0.98	0.992
RF	95.51	0.95	0.91	0.93	0.94	0.95	0.964
GBM	98.95	0.99	0.99	0.99	0.99	0.81	0.991
KNN	95.17	0.98	0.94	0.98	0.92	0.84	0.953
XGB	99.33	0.99	0.99	0.99	0.99	0.93	0.996
Mean of NLC	97.41	0.98	0.96	0.97	0.97	0.9	0.979
Absolute Increase (%)	11.67	8	11	10	12	11	5.5

The evaluation was conducted using performance metrics including accuracy, sensitivity (recall), precision, F1 score, specificity, MCC, and AUC to provide a comprehensive understanding of the classifiers’ effectiveness. The results, presented in Table 9, clearly demonstrate that NLC outperformed LC. The mean accuracy difference between the NLC and LC across all classifiers was 11.67%, while the mean AUC difference was 5.5%. These findings further validate our hypothesis that NLC generalizes better over new datasets due to the nonlinear nature of the intrusion. This analysis provided crucial insights into the comparative performance of LC and NLC in an unseen scenario, contributing to the validation of hypotheses and enhancing the reliability of IDS.

Effect of Training Data Size on the Performance of LC and NLC

To conduct the experiment, we investigated in depth how the size of the training data impacts the performance of our models. The results of our analysis, shown in Table 10, indicate a gradual decrease in performance metrics across multiple cross-validation protocols: K10 (default), K5, and K2. We computed the average mean accuracy and AUC for ten ML classifiers (Five LC and Five NLC) on Dataset-5 (MixMatching).

Table 10. Performance metrics of all ML models on different cross-validation protocols.

	Cross-validation Results				Absolute difference
	K2	K4	K5	K10	D1 = K10-K5	D2 = K10-K4	D3 = K10-K2
Accuracy	86.12	88.61	91.03	91.57	0.54	2.96	5.45
AUC	0.893	0.901	0.936	0.952	1.6	5.1	5.9
Recall	0.859	0.9	0.925	0.94	1.5	4	8.1
Precision	0.855	0.886	0.892	0.909	1.7	2.3	5.4
F1-Score	0.866	0.89	0.911	0.924	1.3	3.4	5.8

D1: Difference between K10 and K5 cross-validation protocols.

D2: Difference between K10 and K4 cross-validation protocols.

D3: Difference between K10 and K2 cross-validation protocols.

We observed that the mean accuracy dropped from 91.57% when using the K10 protocol to 86.12% when using the K2 protocol, representing a decrease of 5.45%. Similarly, the AUC decreased from 0.952 with the K10 protocol to 0.893 with the K2 protocol, signifying a decrease of 5.9%. However, even with a reduced training data size in the K2 (50:50) validation protocol, our ML model’s metrics did not decrease significantly, demonstrating the reliability of our approach. These findings validate that our framework can effectively train for intrusion detection, even when there is limited availability of training data.

Performance Evaluation

As part of the performance evaluation (PE), ROC curves were generated to visualize the performance of the classifier models, depicting their performance across different thresholds. The ROC curves offer a graphical representation of the model’s performance and are useful in assessing its strengths and weaknesses. This evaluation provides valuable insights and identifies areas for improvement in the system. To ensure the reliability and significance of the predicted data, various statistical tests were conducted. Furthermore, a power analysis was performed using our top XGB model to calculate the required sample size, which helps in measuring the precision of the system. This analysis aids in determining the system’s effectiveness and accuracy.

Receiver Operating Curves and Charts

The ROC curves provide a comprehensive analysis of the performance of the LC and NLC classifiers across different cross-validation protocols (K2, K4, K5, and K10). Figure 2 displays the ROC performance of LC, including LR, RR, NB, PERPT, and LDA. Figure 3 presents the ROC performance of NLC, such as DT, RF, GBM, KNN, and XGB. These ROC graphs validate the second hypothesis, demonstrating the effectiveness of training on one dataset and testing on another (WUSTL-EHMS and UNSW-NB15). Additionally, a mix-match operation was performed, combining both the datasets. Among the LC, LR achieved the highest AUC of 0.976, while XGB performed the best among the NLC with an AUC of 0.996, as shown in Figures 2 and 3.

Figure 2. ROC curve for linear classifiers under unseen analysis.

Figure 3. ROC curve for NLC under unseen analysis.

Furthermore, the statistical significance of the results was assessed by computing p-values for all nonlinear models, revealing p < .001. This indicates a high level of confidence in the observed differences between the models and strengthens the validity of the results. Bar charts are an effective visualization technique for presenting information from tables. In Figure 4, the accuracy of both LC and NLC is depicted for the WUSTL-EHMS dataset.

Figure 4. Accuracy of LC and NLC models on WUSTL-EHMS dataset.

The accuracy values range from 69.23% for LDA to 99.32% for GBM, indicating significant improvements in model accuracy. Similarly, in Figure 5, the accuracy of the models is displayed for the UNSW-NB15 dataset. The accuracy values range from 88.92% for PERPT to 99.98% for DT, demonstrating substantial increases in model accuracy. These bar charts provide a clear visual representation of the accuracy performance of the classifiers on the respective datasets.

Figure 5. Accuracy of LC and NLC models on UNSW-NB15 dataset.

Analysis of Reliability and Robustness Using Statistical Tests

The stability and reliability of the top-performing XGB model were evaluated across five datasets using five statistical tests: the Paired t-test, ANOVA test, Z-test, Friedman test, and Wilcoxon test. Paired t-test is a parametric statistical test that compares the mean difference between two paired samples to determine if it is statistically significant. It was used in this study to compare the performance of the XGB model in an unseen analysis. This comparison provided insights into the superiority of NLC. ANOVA test, (Analysis of Variance), is a statistical test that assesses the equality of means across multiple groups or datasets (Pathak and Pathak 2020). In this study, the ANOVA test was used to evaluate the significance of the performance metrics obtained from the XGB model across the different datasets. By monitoring the p-value of the ANOVA test, which needed to be less than 0.01 (p < .0001), the researchers ensured a high level of statistical significance. The Z-test is a standard statistical test used to validate if there is a statistically significant difference between a sample mean and a population mean. In this study, the Z-test (Elhamahmy, Elmahdy, and Saroit 2010) was employed to assess the statistical significance of the p-value obtained from the ANOVA test, providing further validation of the results. The Friedman test (Zhang and Liu 2022) is a non-parametric statistical test that compares the rankings of multiple related samples. It was used in this study to assess the significance of performance metrics across different datasets, taking into account the relative performance of the XGB model in a non-parametric manner.

Wilcoxon test (Silva et al. 2021), also known as the Wilcoxon signed-rank test, is a non-parametric statistical test used to determine if the median difference between two paired samples is statistically significant. In this study, the Wilcoxon test was employed to compare the performance of the XGB model in an unseen analysis, providing additional evidence of the superiority of NLC. The results of these tests, presented in Table 11, demonstrated the performance consistency of the XGB model and provided evidence of its stability as a reliable measure in intrusion detection. The statistical significance of these tests further confirmed the reliability and consistency of the XGB model across the different datasets.

Table 11. Statistical tests p-values of the XGB model using cross-validation protocol.

Statistical p-values for particular datasets using XGB Model
Dataset	Paired t-test	ANOVA test	Z test	Friedman test	Wilcoxon test
Dataset 1	p < .01	p < .01	0.0025	0.0196	0.0026
Dataset 2	p < .01	p < .003	0.0022	0.0000	0.0000
Dataset 3	p < .0001	p < .0001	0.0023	0.0030	0.0000
Dataset 4	p < .0001	p < .0001	0.0019	0.0029	0.0000
Dataset 5	p < .0005	p < .0001	0.0025	0.0008	0.0000

Power Analysis

In order to increase the data appropriately, we conducted a power analysis to identify the minimal sample size required for precisely predicting a population proportion. The sample size calculation formula, denoted by the symbol ${\mathrm{n}}_{\mathrm{s}}$, is as follows:

(7) ${\mathrm{n}}_{\mathrm{s}}=\left[{\left({\mathrm{z}}^{\ast }\right)}^2\times \left(\frac{\tilde{}\mathrm{p}\left(1-\tilde{}\mathrm{p}\right)}{\mathit{Mo}{E}^2}\right)\right]$(7)

Here, $\tilde{}\mathrm{p}$ is the estimated percentage of the feature in the population, $\mathrm{z}\ast$ denotes the Z-score associated with the chosen confidence level, and MoE denotes the margin of error. Half of the breadth of the confidence interval was used to compute the MoE. We settle on a percentage of 0.5 and a confidence interval of 95% for our experiment as shown in Table 12.

Table 12. Power analysis for five datasets using the XGB model.

Power Analysis for five datasets using XGB Model
Dataset	Original Size	Augmented Size	Z* Score	Confidence Interval	Sample Size
Dataset 1	16318	28544	1.96	0.4861–0.5120	5727
Dataset 2	82332	128600	1.95	0.4854–0.5099	6335
Dataset 3	16318	142720	1.64	0.6043–0.6104	72282
Dataset 4	82332	453320	1.95	0.5231–0.5293	98920
Dataset 5	39460	78920	1.95	0.4705–0.4775	77602

The analysis of the intrusion detection datasets, namely WUSTL-EHMS and UNSW-NB15, involved a comprehensive examination of the calculated sample sizes on the best-performing XGB model. The results obtained indicated that the sample sizes utilized in the study surpassed the minimum sample size determined through power analysis. This observation signifies that the research incorporated an abundant number of observations to attain the desired level of statistical power and ensure precise classification during the intrusion detection process. By exceeding the required sample size, the study reinforces the strength and reliability of the findings, while enhancing the representativeness of the data, thereby enabling more accurate generalizations about the population.

In the context of intrusion detection, a larger sample size facilitates a more comprehensive analysis of network traffic patterns, user behaviors, and system logs, leading to a more dependable detection and classification of potential threats. Furthermore, a larger sample size helps reduce the margin of error and enhances the precision of statistical estimates. By ensuring that the sample size surpasses the calculated minimum value, the research study demonstrates a proactive approach toward achieving statistical significance and minimizing the likelihood of type II errors (false negatives). This ensures that the findings of the study are robust and reliable, thereby enhancing the overall quality and credibility of the intrusion detection analysis.

Discussions

Principal Findings

After conducting extensive research, we obtained valuable insights and reached conclusions related to our research problem: (i) To test our hypotheses, we developed a total of ten ML classifiers, consisting of five LC and five NLC. (ii) Our experimental analysis involved the use of five pre-processed datasets, which included two intrusion datasets, two unseen datasets, and one Mix-Match dataset. (iii) We implemented a novel quality control phase in our system to improve the classification of network data and detect intrusions more effectively. (iv) Our findings indicate that NLC outperformed LC, resulting in a significant increase in mean accuracy by 22.26% and a 20.3% increase in AUC on the WUSTL-EHMS dataset. Similarly, on the UNSW-NB15 dataset, NLC exhibited a substantial advantage over LC, with accuracy and AUC improvements of 5.5% and 2.3%, respectively. (v) In the unseen analysis, NLC demonstrated superior performance compared to LC, achieving a mean accuracy of 21.9% and an AUC of 21.8% when trained on WUSTL-EHMS and tested on UNSW-NB15. Additionally, when trained on UNSW-NB15 and tested on WUSTL-EHMS, NLC exhibited a significant mean accuracy increase of 23.68% and an AUC of 17.2%. (vi) Furthermore, when utilizing a mixed dataset of WUSTL-EHMS and UNSW-NB15, the NLC classifiers showed an average accuracy difference of 11.67% compared to LC, with a mean AUC difference of 5.5% across all classifiers. (viii) Finally, we evaluated the impact of training data size by implementing cross-validation protocols. (ix) To ensure the reliability and stability of our system, we subjected the classifiers to statistical tests. (x) To validate the precision of the AI system and assess its stability with smaller data sizes, we conducted a power analysis on the five pre-processed datasets.

Benchmarking: A Comparative Analysis

Numerous researchers have proposed various techniques and approaches for IDS. In the present research study, the chosen approach has undergone extensive analysis and validation to demonstrate the robustness and effectiveness of intrusion detection when employing NLC. Through rigorous examination and validation, it has been established that the adopted approach yields reliable results in detecting intrusions. By leveraging NLC, the study provides evidence of the approach’s capability to effectively identify and classify intrusions, further strengthening its credibility and demonstrating its superiority over existing methods.

Table 13 presents a summary of ten studies that developed classifiers for intrusion detection. Teng et al. (2018) constructed an Environments ‐ Classes, Agents, Roles, Groups, and Objects (E-CARGO) and Context Aware Intelligent Driver Model (CAIDM) model, which employs SVM and DT models in conjunction. These models were trained using the KDD-1999 dataset, resulting in the highest accuracy of 89.02%. Yihunie, Abdelfattah, and Regmi (2019) employed five ML models for an IDS to assess performance based on precision, recall, and F1 score. However, the NSL-KDD dataset used in this study was not normalized, highlighting a flaw that requires further investigation. Hady et al. (2020) proposed a research work that conducted a comparative analysis of ML models, emphasizing performance metrics such as accuracy and AUC values for model evaluation. The EHMS testbed was used for data collection, providing real-time data that facilitated further study. The WUSTL-EHMS-2020 dataset was employed for training and testing the ML models. Additionally, a 10-fold cross-validation methodology was used for experiment validation, demonstrating a certain level of reliability. Nonetheless, further research is still needed in this area.

Table 13. Benchmarking of studies conducted for intrusion detection.

Year	Author	Objective	Methodology	Models	Dataset	Evaluation metrics	Evaluation	Statistical Test	CVP
2018	Teng et al. (2018)	E-CARGO and CAIDM model creation, implementation and comparative study between models	Using SVM and DT models in association showcasing the use of E-CARGO model and CAIDM model	SVM, DT	KDD-1999	Average Accuracy	89.02%	❌	❌
2019	Yihunie, Abdelfattah, and Regmi (2019)	Applying ML to anomaly-based IDS	Using five ml models to measure performance and experimentation is performed without applying normalization on dataset	SVM, SGD, SM, RF, LR	NSL-KDD	Precision, Recall, F1 score	0.9992, 0.9969, 0.9980	❌	❌
2020	Hady et al. (2020)	Comparative study used for classification of normal and attack samples using ML models and measure its performance	Collecting data from EHMS test-bed then running ml models to compare the performance between the models	RF, KNN, SVM, ANN	WUSTL-EHMS-2020	Accuracy, AUC	90.04%, 0.9298	❌	10-fold cross-validation
2020	Abrar et al. (2020)	Comparative study between different data attributes and ML models	Data pre-processing, identifying feature subsets and then training and testing different ML classifiers	SVM, KNN, LR, NB, MLP, RF, ETC	NSL-KDD	Accuracy	99.48%	❌	❌
2021	Kilincer, Ertam, and Sengur (2021)	Comparative study using ML model on various dataset	Classifying intrusion detection using different datasets and evaluate the results using performance metrics	SVM, KNN, DT	CSE-CIC-IDS-2018, ISCX-2012, NSL-KDD, CIDDS-001, UNSW-NB15	Accuracy, Precision, Recall, F1 Score	0.99, 0.99, 0.99, 0.99	❌	K10
2022	Rastogi et al. (2022)	Performance of supervised ML algorithms	Collect the dataset, transform, training and testing on supervised ML models and analyze performance	KNN, SVM, DT, RF, GNB, LR	NSL-KDD	Accuracy	98.70%	❌	❌
2022	Baich et al. (2022)	Comparative study between ML models for intrusion detection	Using ML models and differentiating on the basis of results obtained	DT, SVM, NB, RF	NSL-KDD	Accuracy, Precision, Recall, F1 Score, MCC	99.55%, 0.99, 0.99, 0.99, 0.98	❌	❌
2023	Proposed	Comparative study between LC and NLC	Using statistical tests, performance metrics and cross-validation for getting which classifiers are robust for intrusion detection	LR, RR, NB, PERPT, LDA, DT, RF, GBM, KNN, XGB	WUSTL-EHMS-2020, UNSW-NB15	Accuracy, Sensitivity (recall), Precision, F1 Score, Specificity, MCC, ROC	98.62%, 0.99, 0.99, 0.99, 0.98, 0.99, 0.997	✓	K2, K4, K5 and K10

*CVP: Cross-validation Protocol; ACC: Accuracy (%); ML: Machine Learning; LDA: Linear Discriminant Analysis; LR: Logistic Regression; RR: Ridge Regression; NB: Naive Bayes; PERPT: Perceptron; DT: Decision Tree; GBM: Gradient Boosting Machine; KNN: K-Nearest Neighbors; RF: Random Forest; XGB: Extreme Gradient Boosting; MLP: Multilayer Perceptron (Neural Network); SVM: Support Vector Machine; SGD: Stochastic Gradient Descent; K#: cross-validation protocol having the ratio of training: testing data sets; K2: 50%:50%; K4: 75%:25%; K5:80%:20%; K10: 90%:10%.

Abrar et al. (2020) conducted research on intrusion detection using a different set of attributes to train and test ML models, using the NSL-KDD dataset. The research included data pre-processing, which improved the dataset’s quality and aided ML models in better-identifying feature sets. Kilincer, Ertam, and Sengur (2021) employed DT, KNN, and SVM models on various datasets, including CSE-CIS-IDS-2018, ISCX 2012, NSL-KDD, CIDDS-001, and UNSW-NB15. These models demonstrated impressive performance metrics, with accuracy, precision, and recall values of 0.99. However, the study lacked unseen analysis, which is essential for evaluating the models’ ability to generalize to new and unseen data. This raises concerns about the possibility of overfitting in the reported results. Rastogi et al. (2022) proposed a research work that utilized the NSL-KDD dataset. The paper focused on transforming the data and performing performance evaluation based solely on accuracy as the measurement metric. However, the study lacked statistical analysis and validation of the dataset, which could impact the accuracy of measurements. Baich et al. (2022) presented a comparative study between different ML models using the NSL-KDD dataset. The study employed accuracy, precision, recall, F1 score, and MCC as performance metrics for model evaluation. However, these studies lacked quality control, statistical tests, and comprehensive validation for robust model evaluation.

The proposed study aims to overcome the limitations of the aforementioned works. It conducts a comparative study between different ML classifiers and utilizes two datasets, WUSTL-EHMS-2020 and UNSW-NB15, for training and testing. The proposed work incorporates five LC and five NLC with good quality control applied to the datasets. The models are evaluated using accuracy (99.62%), sensitivity (recall) (0.99), precision (0.99), F1 score (0.99), specificity (0.99), MCC (0.99), and AUC (0.997) in nonlinear scenario on UNSW-NB15 dataset. These values are more efficient due to the statistical tests performed, considering a significance p-value <.001. The unseen analysis is conducted by training and testing on combinations of datasets and employing mix-matching techniques, making it unique in terms of evaluating ML classifiers. Additionally, K2, K4, K5, and K10 cross-validation methods are applied, and performance metrics are calculated for each cross-validation, providing an overall assessment of model performance.

By surpassing the limitations of previous research works and addressing the evaluation of ML classifiers in a unique and comprehensive manner, this study provides a reliable and robust system for intrusion detection.

Special Note on Nonlinear Classifier for Intrusion Detection

The advantage of implementing NLC for intrusion detection datasets arises from the complex and intricate relationships exhibited by cyber threats and network activities. Linear models are unable to effectively capture these relationships, given the nonlinear nature of intrusion data and logs. In intrusion detection, the dataset encompasses various factors, including network traffic patterns, user behaviors, and attack signatures, which interact in nonlinear ways. Consequently, LC fails to capture the intricate relationships and nonlinearity present in the data, resulting in limited detection capabilities.

NLC, on the other hand, are specifically designed to handle complex patterns and nonlinear relationships. By employing nonlinear decision boundaries and flexible modeling techniques, these classifiers can effectively capture and analyze the intricate dynamics of the intrusion detection dataset. Leveraging the capabilities of NLC enables IDS to improve their accuracy, sensitivity, and specificity in detecting various types of intrusions, including both known and unknown attacks. This advantage allows for more robust and comprehensive intrusion detection, enhancing the security and reliability of network environments.

Incorporating NLC in IDS enhances the system’s ability to accurately identify and classify anomalous activities, thereby strengthening the overall security of computer networks.

Strength, Weakness, and Extensions

The research findings demonstrate a novel approach to intrusion detection by utilizing ten ML models, comprising five LC and five NLC. The study successfully validates four novel hypotheses, focusing on linear and nonlinear aspects, unseen analysis, and the size of the training data. The approach proves effective in generalizing the models’ performance, showcasing superior results in unseen scenarios. Statistical tests conducted on the classifier’s outcomes and the calculation of sample size contribute to the generalization of the findings The approach significantly improves the performance of the models across all datasets, leading to a notable enhancement in classification accuracy within the IDS. These findings highlight the potential of the proposed approach to greatly improve the effectiveness and reliability of intrusion detection in various network environments.

However, the intrusion detection scheme we have developed may encounter challenges in classification accuracy and AUC when applied to a broader range of scenarios with a more complex network architecture. This is primarily due to the limited size of the training data available for our model. Furthermore, our intrusion detection scheme lacks feature extraction as it does not utilize DL models. This absence of feature extraction could limit the model’s capability to classify accurately and diminish its robustness and reliability.

In the future, the focus should be on utilizing feature extraction techniques to enhance the performance of ML models in intrusion detection. By employing these techniques, followed by proper training, testing, and validation of the models, it is possible to achieve better accuracy in detecting intrusions (Saheed et al. 2022; Singh et al. 2023; Yan and Han 2018). Additionally, collecting and incorporating a larger volume of data can further improve the system’s ability to learn and identify flaws more effectively. Cloud-based deep learning (DL) models (Saba et al. 2016, 2017; Suri 2020; Suri et al. 2022) that employ a federated learning-based paradigm, comprising a GUI, logic layer, and persistence layer, have shown promise in intrusion detection. Implementing such models as IDS (Modi et al. 2013; Roschke, Cheng, and Meinel 2009; Yassin et al. 2012) could yield positive results. The utilization of Explainable AI techniques (Zebin, Rezvy, and Luo 2022; Zhang et al. 2022) can also enhance interpretability in this paradigm. By employing these techniques, the IDS system becomes more transparent, allowing security professionals to better understand the decision-making processes of the models and providing insights into the detected intrusions. These advancements will contribute to more effective and reliable detection of intrusions in versatile network environments.

Conclusion

The proposed research work presents a novel approach to intrusion detection through the comprehensive utilization of machine learning models. Intrusions are classified using ten distinct models, comprising five LC and five NLC. Our findings demonstrate the superiority of NLC over LC in intrusion detection. Furthermore, we assess the generalizability of our scheme by evaluating the performance of the models in unseen analysis. To validate the reliability and significance of our approach, we conduct robust statistical tests and power analyses. In conclusion, the proposed scheme, with its utilization of machine learning models and the demonstrated superiority of nonlinear classifiers, offers a promising approach to effectively mitigate intrusions and enhance the security of computer networks.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

Due to its proprietary nature, supporting data cannot be made available openly.