Malicious traffic classification using hybrid heuristic-payload based technique with machine learning

Abstract

With the emergence of new malware variants, the existing intrusion detection are no longer effective as most of the methods are not capable of detecting unknown traffic. In this paper, we proposed malicious traffic detection using hybrid heuristic-payload based technique with machine learning classification. Heuristic classifier classifies network traffic to two classes, malicious and non-malicious. 5-gram features are extracted from the traffic payload as the training set and trained to generate the classifier model. The performance of the classification in terms of accuracy and efficiency for different algorithms are analyzed and the results indicate that Random Tree algorithm shows the best performance. The proposed method is benchmarked with classification using payload based only and signature based malware detection, Snort. Results show that the proposed method is 16.06% and 6.48% more efficient than classification using payload based only for cross validation and supplied test set test options, respectively and two times more accurate than Snort. The proposed method is found effective as it has high accuracy and good efficiency compared to payload based only and signature based malware detection.