Applying unsupervised system-call based software security techniques for anomaly detection

Abstract

System call analysis is a technique intended for detecting malware. The above method helps in achieving better detection accuracy. Thus, machine learning (ML) techniques are used for this evaluation. This paper discusses unsupervised ML techniques to detect malware. Our proposed detector monitors the software and marks them anomalous or benign based on their behavior. Experimental results provide performance statistics based on the true positive rate at a low false positive rate. As we got considerable accuracy in some models, there is scope for designing an anomaly detection system centered on unsupervised learning. We illustrated how models performed against various malware samples when executed on benign hosts and testbeds. We included a case study to mitigate the adversary attack on the anomaly detection system.

Subject Classification:

• 68M25

Keywords:

• System call
• Unsupervised machine learning
• Anomaly detection system
• Adversary attack