Network Intrusion Detection Techniques using Machine Learning

CREDITS

Name	Email Address
Abdul Aziz	[email protected]
Murtaza Ahmed Siddiqi	[email protected]

ABSTRACT

 With the rapid development and an increasingly vital role of technology in the interconnected world, security is among the most significant concerns. Among numerous solutions, Intrusion detection systems (IDS) is considered one of the optimum system for detecting different kind of attacks. In this paper, we have tried to present a comprehensive study on Network Intrusion detection system (NIDS) techniques using Machine Learning (ML). Based on our study of recent research we have highlighted the latest techniques of NIDS using ML techniques and approaches, the common attacks that they can detect, and their issues. The purpose of this study is to share the collected information in a way that is more comprehensive and can provide an overview of the current standing of ML-based NIDS.

1.INTRODUCTION

At present, the use of digital technologies in every area specifically in business is rapidly growing. Alongside the positive impact and growth of the internet for global communication, one cannot ignore its dark side. With the progression of the interconnected device, attacks are also increased; networks are always at risk by the threat actors. Maintaining security is among the most essential aspects of any organization. The three paramount pillars of network security [1] are shown in figure 1.

Figure 1: Basic pillars of Network Security

Although it is difficult to estimate the exact financial loss caused by cyber-attacks, an average estimation can be seen in figure 2 [2]. An example of such an attack was the DYN cyber-attack in 2016; DYN was a Distributed Denial of Services (DDOS) based attack. The attack targeted the systems operated by Domain Name System (DNS) providers and the VPN Filters. As a result, malware successfully infected around 500,000 routers and Storage Area Network (SAN) devices worldwide [1].

Figure 2: Organizational financial loss due to Cyber-attacks

The orthodox approach for securing a computer network is to define policies, design security mechanisms (i.e. Firewall, Virtual Private Network, authentication and authorization mechanism etc.). However, such approaches have been around for a period of time and the attackers have found numerous ways to bypass such approaches. Due to the mentioned reason, a need for improved security and monitoring mechanism was required for which the IDS is considered as one of the most effective and widely used approach in today’s world of network security. The IDS can be broadly classified into two types, Network-based Intrusion Detection System (NIDS) and Host-based Intrusion Detection Systems (HIDS).

Type	Advantage	Disadvantage
NIDS	NIDS generally exists at a point in the network where it can monitor both incoming and outgoing traffic.  Its the environment independent and does not affect host performance	IT can affect network bandwidth; also, it cannot be able to detect events occurring at different places at the same time. Adversaries may incapacitate the software running of IDS making it unreliable. It cannot analyze the encrypted channel traffic and have limited visibility on host machine
HIDS	It’s a Locally host based. Inside visibility of host in case of attacks either its successful or not. Such systems can analyze the encrypted communications	Each host on a network needs to have it installed and this can degrade the performance of the system as these resource intensive

Table 1: Different between NIDS and HIDS [3]

The mentioned IDS types in table 1 have two basic methods of intrusion detection, Signature-based and Anomaly-based detection [4]. In signature-based detection, all incoming packets are forwarded to a database containing the signatures of known malicious patterns also known as indicator of compromises. While in anomaly-based detection, network traffic is first analyzed to establish a model of normal behaviors of the network. This model includes information such as bandwidth utilization, protocols, the port used, IP addresses, etc. Then network traffic is compared with the established model to detect any deviation or anomaly. As the attacks grow more sophisticated and difficult to detect, most of the vendors and security experts are seeing ML-based IDS as a solution to current security requirements. ML-based IDS are traditionally trained by the use of predefined datasets from the emulated or simulated environment set-up [5]. The Learning methods of an ML-based IDS can be differentiated as shown in table 2.

 

Learning Type	Description	Algorithms
Supervised	The desired result in supervised learning is already known. The labeled dataset is provided to the algorithm to learn and construct a model based on the provided dataset. Supervised algorithms can be categorized into classification and regression-based algorithms.	·       Decision Trees, ·       K-Nearest Neighbor ·       Linear Regression ·       Support Vector Machine ·       Neural Networks
Unsupervised	The dataset provided in unsupervised learning does not contain any label. The algorithm learns itself and discovers the hidden patterns to identify each data type. An unsupervised algorithm can further be classified into the clustering algorithm, outlier detection, Association Rule Mining, and Auto encoders.	·       Apriori ·       K-means clustering and other association rule mining algorithms.
Reinforced	Reinforced algorithms learn by the feedback methods and experiences. One can say that it’s a trial and error-based learning.	·       Q-Learning ·       Deep Adversarial Networks ·       Temporal Difference
Semi-Supervised	Semi-Supervised learning algorithms learn using both labeled and non-labeled data. Such approach is used when extracting useful features or when the labeling data is difficult (i.e. CT scan, MRIs where experts can identify some points to identify a medical condition but may not be able to identify every indicator).	·       Generative models ·       Low-density separation ·       Graph-based methods ·       Heuristic approaches

Table 2: Summary of ML algorithm types [6] [7]

Highlighted ML algorithms and approaches in table 2 can further be divided into two categories; Classical Learning and Deep Learning. The recent increase of ML-based solutions gave a way to Neural Network (NN) based approaches. Generally, DNN based approaches come with a cost of heavy computations, however with the recent advancements in technology computational requirements are not a major concerns [8]. The rest of the paper is itemization into the following sections. Section II covers a basic introduction to the ML framework for IDS, providing readers with an overview of how ML-based IDS and specifically NIDS work. Section III covers related work, providing the insight of recent work done in ML-based NIDS. Section IV covers an in-depth analysis of recent techniques for IDS. In Section V, a recommendation for an improved NIDS is suggested based on the studied material. Section VI will conclude the paper.

2.MACHINE LEARNING FRAMEWORK FOR IDS AND NIDS

As the use of IT increases, intruders are launching sophisticated attacks on the networks and systems to gain unauthorized access. So to detect such malicious attacks, IDS are used. IDS are placed in the network according to the planning and topology for effective detection of attacks, it requires the proper assessment of physical and logical IDS that can detect an intrusion based on the attack signature or by analyzing the behavior of network traffic [9]. The basic functions of the IDS can be seen in figure 3.

Figure 3: Basic IDS functions

However, the NIDS is a type of IDS that is specifically designed to analyze and the detection of the network-specific attacks. NIDS is commonly deployed with an aggregation switch, where the overall picture of the network is observable.  A common deployment of IDS is illustrated in Figure 4.

Figure 4: General NIDS System model

The NIDS alarms the administrator and generates the event logs in case of malicious activity detection. In the recent times, the hybrid NIDS approach is used due to its effective detection. A hybrid NIDS is a combination of both signature and anomaly-based detection [9]. Table 3 gives an overview of signature, anomaly, and hybrid NIDS.

NIDS Type	Description
Signature-based	It is Suitable only for known threats as the main idea behind signature-based IDS is the same as a virus scanner. Signature-based IDS searches for known malicious patterns to identify them.
Anomaly-based	Such IDS identify changes in network traffic normal behavior. It works by identifying known and legal traffic on the network.
Hybrid	Hybrid is simply a combination of both Signature and Anomaly-based IDS. As both of these approaches have their pros and cons.

Table 3: NIDS threat detection approaches

3.RELATED WORK

With the advancement in technology, cybersecurity is becoming an integral part of our daily online activities. The increased ratio in attacks is also becoming more powerful and difficult to mitigate. Security researchers and hackers are always in a war to win. The Mitigation of sophisticated attacks requires high-end hardware or a well-developed software-based system. Existing tools for detection and prevention requires constant maintenance to counter recent attacks. Maintaining mentioned systems also include updating and checking logs on daily basis, which is quite hectic for security experts. Since the internet world is advanced with technology, the attacker’s strategies are also changed which ultimately makes the traditional tools vulnerable. Such liabilities allow intruders to bypass and evade security systems and allowing the intruders to perform malicious activities [2] [3]. R. Thornton [4] have proposed a model that uses Machine Learning (ML) and Deep learning (DL) approaches to detect unknown attacks. Authors discussed various ML & DL techniques that comes under supervised and unsupervised learning techniques. In the papers [3][5][6], authors have proposed a model in which different data sources such as logs, packets flow, and sessions are collected and presented to ML algorithms. Although, to achieve high accuracy a proper standardized data-set must be available. As per the different reviewed papers for the study, 2017 was the year in which maximum number of publications in journals and conferences were on DL and ANN. This trend is still ongoing with increasing number of publications related to ML or DL techniques. The study shows that the most broadly used methods for NIDS are DL and ANN. While the most common dataset for NIDS evaluation are KDD199 [24] and NSL KDD [25] respectively.

Based on these research studies, R. Thornton[4] highlighted that research in the field of ML based security solutions will increase in the coming days. Due to the increase of ML based security solutions, industrial and academic researchers will be able to further optimize and enhance ML to the next level. The future of NIDS would be to develop and use these approaches to detect new and emerging threats. As every year is a new year for the threat actors to attack more powerfully on networks using different techniques and strategies. Attack vectors, surfaces are changing with every passing year and as a result of this protecting the resources is gaining more importance as well in the era of technology. Using traditional techniques to detect and protect against malicious attacks is being now useless. In the paper [6], the author has justified that the world is moving towards Artificial Intelligence (AI), which will highlight the use of AI with automated tools like NIDS. Intrusions detection based on signature-based detection are not so effective against modern attacks. Therefore, using Deep Learning and ANN based approaches for NIDS are being increasingly researched since 2017.

Different authors have reviewed several techniques that are used in NIDS, including machine learning, deep learning and data mining, such approaches are classified as anomaly based NIDS. However, the most recent study has introduced the hybrid approach that includes anomaly and signature based NIDS. These techniques are used to protect several kinds of different network attacks such as DoS, DDoS, probe attacks and IP Spoofing, etc. Furthermore, deep learning shows effective results in the detection rate of network attacks which can be further improved by using the hybrid technique with the help of machine learning classifiers [7]. Authors in this paper have discussed the use of sandboxing technique. Sandbox uses ML as a tool to secure the network system from the cyber-attacks. Sandboxing along with machine learning helps in the malware detection. If the data is predicted to be malware it is sent to the Sandbox for analysis inside a Sandbox VM. This way it could not infect the original Machine or system on which certain other process are running [8].

In the paper [8], authors have discussed several ML techniques including Bayesian Network, Decision tree, Clustering, ANN, Genetic Algorithm, Hidden Markov Models (HMM), and Inductive Learning. Authors also suggested to use clustering algorithms which work best in anomaly detection.

Authors in the paper [9] proposed a hybrid solution of using both Signature based IDS and Anomaly based IDS. For the detection of anomaly, classifiers generate signatures and store them in to the signature database for the early detection. In this paper authors evaluated the performance of different ML algorithms based on different datasets. As per experimentations, J48 and Random forest achieved the best performance. Based on the experimentations authors have proposed 4 major types of ML based algorithms which could perform very effectively [10]. The authors in this paper have achieved better accuracy than previous intrusion detection works. Most of the previous work performed intrusion detection based on two well-known datasets, KDD-CUPP99 [11] and NSL-KDD [12]. Further, authors have now started the use of machine learning and the deep learning algorithms which are specifically used for the NIDS for achieving the better performance and reduces false positives [13] [14] [15] [16].

This paper is survey-based in which various studies are reviewed to highlight the most recent techniques that can be used in NIDS. Machine learning is the super-set of deep learning which is considered one of the useful methods for detecting the anomalous behaviors in intrusion detection. The databases used for the papers are restricted to IEEE and scope up to the past 4 years 2017-2020. The Papers used in this study are restricted to the English language, the search text used for the papers are; “Intrusion Detection”, “Network Intrusion Detection”.

4.RECENT TECHNIQUES FOR INTRUSION DETECTION

The IDS by function can be divided into types; signature-based IDS and the anomaly-based IDS [17]. The signature-based IDS requires the signature patterns available in its signature database to be compared with the packet signature received by the sensor for the intrusion detection, for this type of system it needs to be up to date with due respect to time, and also its only effective for the known attacks. However, on the other side, the anomaly detection based Intrusion detection is effective for the detection of unknown attacks or intrusions.

It relies on the behavior of the system and compares the system’s normal behavior with the deviated behavior if attacked by any threat actor. [17] The set of features are used to identify the network connections such as; service, protocol, number of login attempts, packets per flow, bytes per flow, source address, destination address, source port, destination port, and others. The features values are recorded by the model and any deviation in recorded values will be marked as anomalous by the anomaly detection engine. Techniques in anomaly detection can be categorized into three types; Machine Learning, Statistical Techniques and Finite State Machine. This paper focuses on Machine learning-based techniques, for the detection of anomaly semi-supervised and unsupervised machine learning algorithms can be used such as; Self Organizing Map (SOM), Neural Network [18], clustering and unsupervised algorithm [19], and the Support Vector Machine [20]. Table 4 differentiates the three techniques of anomaly detection.

NIDS Type	Description
Machine Learning	Machine Learning approaches provides the capability to learn and detect the attacks, such approaches learn from the experience and the datasets given to the models which is built by the used approaches. The attacks are detected by analyzing the behavior and patterns of the traffic against the model it build.
Statistical Learning	These techniques use statistical properties to build the normal profile of transactions via using ‘mean’ deviation and others. Such techniques don’t require any prior knowledge of attacks to detect the new zero day attacks. Statistical approaches determines the deviation between observed and normal traffic.
Finite Sate Machine	It’s a mathematical model with finite number of states at some given time FSM is a mathematical model of computation with finite number of states at any given time, these states are initial, possible and attack states. The FSM model keeps the track of events in directed graphs so that it can be used to identify the attack.

Table 4: Anomaly detection techniques

Table 5 highlights the most recent published work on machine learning-based techniques. The table highlights data sets used, ML approach and the attacked detected.

Serial	Paper	ML approach	Dataset	Detected attacks
1	[21]	SVM	KDD99	DOS, Probe, U2R, R2L
2	[22]	Decision Tree, C4.5	RLD09, KDD99	DOS, Probe
3	[23]	Neural Network	KDD99	DOS, Probe, U2r, R2L
4	[24]	MOGA	ISCX	DOS,DDOS, Bruteforce,SSh
5	[25]	Neural Fuzzy and GA	KDD99	DOS, Probe, U2R, R2L
6	[26]	K-Mean and NB	ICSX	DOS,DDOS, Bruteforce,SSh
7	[27]	K-means, Nero Fuzzing and SVM Classifier	KDD99	DOS, Probe, U2R, R2L
8	[28]	Bayesian Clustering	KDD99	DOS, Probe, U2R, R2L
9	[29]	Logistic Regression	UNSW-NB15	DOS, Fuzzer
10	[30]	SVM	UNSW-NB15	DOS, Fuzzer

Table 5: Techniques, datasets & detected attacks

5.RECOMMENDATIONS

In this paper, the highlighted techniques are the most effective although not for every attack but are well known in detecting the attacks. However, the security researchers can continue this study and achieve better results by adding more of the Deep Learning algorithms. Table 5 highlights the pros and cons of the NIDS approaches. This paper also revealed that 43 papers were searched from the selected databases like ACM, IEEE, Google Scholar, and other containing the search strategy “Forward and Backward Snowballing. Further, the construction of search strings is as per the architecture such as ANN, Recurrent Neural Networks (RNN), DL, Deep Networks, and the synonyms of NIDS like Intrusion Detection, NIDS, and Anomaly-based intrusion detection.

NIDS Approach	Pros	Cons
Signature Based	·       Its lightweight with simple implementation due to the low computation required. ·       The false-positive rate ratio is very low. ·       It is effective for the know attacks.	·       The zero-day attacks are hardly detectable of new attacks. ·       It Can affect high bandwidth network speed due to the signature check from database.
Anomaly Based	·       It is based on model and behavior analysis ·       It learns from the attacks. ·       It Can detect zero-day as well as new attack patterns.	·       Training required for normal traffic baseline. ·       False-positive rate can be high. ·       Resource expensive operation. ·       Complex implementation.
Hybrid	·       It Combines the strength of both host and network IDS for the high accuracy in results. ·       It provides more effective results than others.	·       It takes long time to analyze the traffic. ·       It increases the processor utilization extensively.

Table 6: NIDS approaches pros and cons

6.CONCLUSION

The increasing growth in the cyber-attacks on Networks and Hosts have disrupted many organizations and individuals as well. Nevertheless, Security researchers are doing their best to detect such attacks. Despite several solutions available in the market are not that capable of detecting attacks effectively. This paper has highlighted the most recent techniques and the security attacks along with data-sets used which are then compared against the attacks which revealed that a technique providing the best performance for any attack, may or may not perform or give the same performance. Not every method can be best fit for all different types of techniques due to the continuous increase in the cyber-attacks in which characteristics are changed every time hence detection of new attacks of all kinds at the Network IDS is impossible. This is based on the various factors like features of each attack and the use of classifiers affects the performance positively. Furthermore, there is still a need for new data-sets for better performance in the future.

REFERENCES

1. Magán-Carrión, D. Urda, I. Díaz-Cano and B. Dorronsoro, “Towards a Reliable Comparison and Evaluation of Network Intrusion Detection Systems Based on Machine Learning Approaches,” MDPI Appl. Sci, vol. 10, no. 5, p. 1775, 2020.
2. Brook, “What’s the Cost of a Data Breach in 2019?,” Digital Guardian, London, 2019.
3. K. G. a. H. S. G. L. Mehra, “An effectual & secure approach for the detection and efficient searching of Network Intrusion Detection System (NIDS),” in 2015 International Conference on Computer, Communication and Control (IC4), Indore, 2015.
4. Thornton, “AT&T Business and Cybersecurity,” AT&T, 20 july 2020. [Online]. Available: https://cybersecurity.att.com/solutions/intrusion-detection-system/ids-explained. [Accessed 21 july 2020].
5. S. Nour Moustafa, “The Evaluation of Network Anomaly Detection Systems: Statistical Analysis of the UNSW-NB15 Data Set and the Comparison with the KDD99 Data Set,” Information Security Journal: A Global Perspective, vol. 25, no. 1-3, 2016.
6. B. Liu H, “Machine Learning and Deep Learning Methods for Intrusion Detection Systems: A Survey.,” MDPI, Applied Sciences, vol. 9, no. 20, p. 4396, 2019.
7. STH, “Types Of Machine Learning: Supervised Vs Unsupervised Learning,” Software Testing Help, 30 June 2020. [Online]. Available: https://www.softwaretestinghelp.com/types-of-machine-learning-supervised-unsupervised/. [Accessed 22 July 2020].
8. P. M. &. R. J. Drewek-Ossowicka, “A survey of neural networks usage for intrusion detection systems,” Journal of Ambient Intelligence and Humanized Computing (2020), 2020.
9. Ansam , G. Iqbal and P. Vamplew, “Hybrid Intrusion Detection System Based on the Stacking Ensemble of C5 Decision Tree Classifier and One Class Support Vector Machine,” Electronics – mdpi, 17 January 2020.
10. R. Varanasi and S. Razia, “Intrusion Detection using Machine Learning and Deep Learning,” International Journal of Recent Technology and Engineering Regular Issue, vol. 8,no. 4, pp. 9704–9719, 2019.
11. Karatas, O. Demir, and O. K. Sahingoz, “Deep Learning in Intrusion Detection Systems,” 2018 International Congress on Big Data, Deep Learning and Fighting Cyber Terrorism (IBIGDELFT), pp. 113–116, 2018.
12. Sagar Dhende , Dr. R.B. Ingle ,” Survey on Network Intrusion Detection System (NIDS)” International Journal of Innovative Research in Computer and Communication Engineering ,2018
13. Shone, Nathan, Tran Nguyen Ngoc, Vu Dinh Phai, and Qi Shi. ”A deep learning approach to network intrusion detection.” IEEE Transactions on Emerging Topics in Computational Intelligence 2, no. 1 (2018): 41-50.
14. Nandurdikar, Bhakti, and Rupesh Mahajan. ”A Survey on Intelligent and Effective Intrusion Detection system using Machine Learning Algorithm.”2020
15. Siddique, K.; Akhtar, Z.; Aslam Khan, F.; Kim, Y. KDD Cup 99 Data Sets: A Perspective on the Role of Data Sets in Network Intrusion Detection Research. Computer 2019, 52, 41–51.
16. Haider,W.; Hu, J.; Slay, J.; Turnbull, B.P.; Xie, Y. Generating realistic intrusion detection system dataset based son fuzzy qualitative modeling. J. Netw. Comput. Appl. 2017, 87, 185–192.
17. A Detailed Investigation and Analysis of Using Machine Learning Techniques for Intrusion Detection, Preeti Mishra , Member, IEEE, Vijay Varadharajan, Senior Member, IEEE, Uday Tupakula, Member, IEEE,and Emmanuel S. Pilli , Senior Member, IEEE, IEEE COMMUNICATIONS SURVEYS and TUTORIALS, VOL. 21, NO. 1, FIRST QUARTER 2019
18. Kohonen, “The self-organizing map,” Proc. IEEE, vol. 78, no. 9, pp. 1464–1480, Sep. 1990.
19. Casas, J. Mazel, and P. Owezarski, “Unsupervised network intrusion detection systems: Detecting the unknown without knowledge,” Comput. Commun., vol. 35, no. 7, pp. 772–783, 2012.
20. Yang, T. Deng, and R. Sui, “An adaptive weighted one-class SVM for robust outlier detection,” in Proc. Chin. Intell. Syst. Conf., 2016, pp. 475–484.
21. S. Kim and J. S. Park, “Network-based intrusion detection with support vector machines,” in Information Networking. ICOIN 2003 (LNCS 2662), H. K. Kahng. Heidelberg, Germany: Springer, 2003, pp. 747–756.
22. Sangkatsanee, N. Wattanapongsakorn, and C.
23. http://caesar.web.engr.illinois .edu/courses/CS598.S13/slides/philip_IDS_practice.pdf,
24. https://archive.ics.uci.edu/ml/datasets/kdd+cup+1999+data
25. https://www.unb.ca/cic/datasets/nsl.html